Diff 30273

swh/loader/package/pubdev/loader.py

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				import json
				from pathlib import Path
				import sqlite3

				from swh.model.identifiers import CONTENT, SWHID, parse_swhid
				from swh.web.client.client import typify


				class Cache:
				def __init__(self, cache_dir):
				seirlUnsubmitted Done Inline Actions I would like this class to be more pythonic by working as a context manager. For me, the sqlite3 connection should probably happen in an `__enter__()` function, and closing it in an `__exit__()` function. I know you cannot use a with block in the Fuse operation class (a shame...), but at least it keeps the quirky aspect of this solely in the Fuse class. seirl: I would like this class to be more pythonic by working as a context manager. For me, the…
				seirlUnsubmitted Done Inline Actions As discussed, we probably want to have two caches (they can probably both inherit some common behavior from an abstract base if that's helpful). If you do go ahead with two classes, you will be able to use getitem and setitem to enable a more pythonic way of accessing this cache, like `blob_cache[swhid] = X`. seirl: As discussed, we probably want to have two caches (they can probably both inherit some common…
				if cache_dir == ":memory:":
				self.conn = sqlite3.connect(":memory:")
				else:
				cache_path = Path(cache_dir, "cache.sqlite")
				cache_path.parents[0].mkdir(parents=True, exist_ok=True)
				self.conn = sqlite3.connect(cache_path)

				self.db = self.conn.cursor()
				self.db.execute("create table if not exists metadata_cache (swhid, metadata)")
				self.db.execute("create table if not exists blob_cache (swhid, blob)")
				self.conn.commit()
				zackUnsubmitted Done Inline Actions if the sqlite3 module is a well-behaved Python DB module, this could be better written with a contextmanager that commits at the end zack: if the sqlite3 module is a well-behaved Python DB module, this could be better written with a…

				seirlUnsubmitted Done Inline Actions I think it's weird to keep a long-lived cursor in the class, you should probably create a new one whenever you're executing something without the `conn.execute(` shortcuts. I think the class should only hold the connection as its state, not the currently active cursors. seirl:* I think it's weird to keep a long-lived cursor in the class, you should probably create a new…
				def get_metadata(self, swhid: SWHID):
				zackUnsubmitted Done Inline Actions Please add docstring to all public methods, classes and modules. No need to be very detailed for now, oneliners would do for starters. But it's much better to have them since the beginning than having to add all of them later. zack: Please add docstring to all public methods, classes and modules. No need to be very detailed…
				self.db.execute(
				"select metadata from metadata_cache where swhid=?", (str(swhid),)
				)
				cache = self.db.fetchone()
				if cache:
				metadata = json.loads(cache[0])
				return typify(metadata, swhid.object_type)

				def put_metadata(self, swhid: SWHID, metadata):
				zackUnsubmitted Done Inline Actions methods that do not return anything should be explicitly annotated with `-> None`, otherwise the default return value is `Any` (h/t @olasd ) zack: methods that do not return anything should be explicitly annotated with `-> None`, otherwise…
				self.db.execute(
				"insert into metadata_cache values (?, ?)",
				(
				str(swhid),
				json.dumps(
				metadata,
				# Converts the typified JSON to plain str version
				default=lambda x: x.object_id
				if isinstance(x, SWHID)
				else x.__dict__,
				),
				),
				)
				self.conn.commit()

				def get_metadata_swhids(self):
				self.db.execute("select swhid from metadata_cache")
				swhids = self.db.fetchall()
				swhids = [parse_swhid(x[0]) for x in swhids]
				return swhids

				def get_blob(self, swhid: SWHID):
				if swhid.object_type != CONTENT:
				zackUnsubmitted Done Inline Actions We might want to fix this in swh-web-client, making typify optional in return values, as it's really silly to typify there and goes the other way here. Can you file a dedicated task for this? zack: We might want to fix this in swh-web-client, making typify optional in return values, as it's…
				haltodeAuthorUnsubmitted Done Inline Actions Filed in: T2646. haltode: Filed in: T2646.
				raise AttributeError("Cannot retrieve blob from non-content object type")
				seirlUnsubmitted Done Inline Actions While this fix is here temporarily, make it less hard to read by using parentheses, it's not obvious what the "if" applies to here. seirl: While this fix is here temporarily, make it less hard to read by using parentheses, it's not…

				self.db.execute("select blob from blob_cache where swhid=?", (str(swhid),))
				cache = self.db.fetchone()
				if cache:
				blob = cache[0]
				return blob

				def put_blob(self, swhid: SWHID, blob):
				self.db.execute("insert into blob_cache values (?, ?)", (str(swhid), blob))
				self.conn.commit()

View Options

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_Autolinker

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				import json
				from pathlib import Path
				import sqlite3

				from swh.model.identifiers import CONTENT, SWHID, parse_swhid
				from swh.web.client.client import typify


				class Cache:
				def __init__(self, cache_dir):
				seirlUnsubmitted Done Inline Actions I would like this class to be more pythonic by working as a context manager. For me, the sqlite3 connection should probably happen in an `__enter__()` function, and closing it in an `__exit__()` function. I know you cannot use a with block in the Fuse operation class (a shame...), but at least it keeps the quirky aspect of this solely in the Fuse class. seirl: I would like this class to be more pythonic by working as a context manager. For me, the…
				seirlUnsubmitted Done Inline Actions As discussed, we probably want to have two caches (they can probably both inherit some common behavior from an abstract base if that's helpful). If you do go ahead with two classes, you will be able to use getitem and setitem to enable a more pythonic way of accessing this cache, like `blob_cache[swhid] = X`. seirl: As discussed, we probably want to have two caches (they can probably both inherit some common…
				if cache_dir == ":memory:":
				self.conn = sqlite3.connect(":memory:")
				else:
				cache_path = Path(cache_dir, "cache.sqlite")
				cache_path.parents[0].mkdir(parents=True, exist_ok=True)
				self.conn = sqlite3.connect(cache_path)

				self.db = self.conn.cursor()
				self.db.execute("create table if not exists metadata_cache (swhid, metadata)")
				self.db.execute("create table if not exists blob_cache (swhid, blob)")
				self.conn.commit()
				zackUnsubmitted Done Inline Actions if the sqlite3 module is a well-behaved Python DB module, this could be better written with a contextmanager that commits at the end zack: if the sqlite3 module is a well-behaved Python DB module, this could be better written with a…

				seirlUnsubmitted Done Inline Actions I think it's weird to keep a long-lived cursor in the class, you should probably create a new one whenever you're executing something without the `conn.execute(` shortcuts. I think the class should only hold the connection as its state, not the currently active cursors. seirl:* I think it's weird to keep a long-lived cursor in the class, you should probably create a new…
				def get_metadata(self, swhid: SWHID):
				zackUnsubmitted Done Inline Actions Please add docstring to all public methods, classes and modules. No need to be very detailed for now, oneliners would do for starters. But it's much better to have them since the beginning than having to add all of them later. zack: Please add docstring to all public methods, classes and modules. No need to be very detailed…
				self.db.execute(
				"select metadata from metadata_cache where swhid=?", (str(swhid),)
				)
				cache = self.db.fetchone()
				if cache:
				metadata = json.loads(cache[0])
				return typify(metadata, swhid.object_type)

				def put_metadata(self, swhid: SWHID, metadata):
				zackUnsubmitted Done Inline Actions methods that do not return anything should be explicitly annotated with `-> None`, otherwise the default return value is `Any` (h/t @olasd ) zack: methods that do not return anything should be explicitly annotated with `-> None`, otherwise…
				self.db.execute(
				"insert into metadata_cache values (?, ?)",
				(
				str(swhid),
				json.dumps(
				metadata,
				# Converts the typified JSON to plain str version
				default=lambda x: x.object_id
				if isinstance(x, SWHID)
				else x.__dict__,
				),
				),
				)
				self.conn.commit()

				def get_metadata_swhids(self):
				self.db.execute("select swhid from metadata_cache")
				swhids = self.db.fetchall()
				swhids = [parse_swhid(x[0]) for x in swhids]
				return swhids

				def get_blob(self, swhid: SWHID):
				if swhid.object_type != CONTENT:
				zackUnsubmitted Done Inline Actions We might want to fix this in swh-web-client, making typify optional in return values, as it's really silly to typify there and goes the other way here. Can you file a dedicated task for this? zack: We might want to fix this in swh-web-client, making typify optional in return values, as it's…
				haltodeAuthorUnsubmitted Done Inline Actions Filed in: T2646. haltode: Filed in: T2646.
				raise AttributeError("Cannot retrieve blob from non-content object type")
				seirlUnsubmitted Done Inline Actions While this fix is here temporarily, make it less hard to read by using parentheses, it's not obvious what the "if" applies to here. seirl: While this fix is here temporarily, make it less hard to read by using parentheses, it's not…

				self.db.execute("select blob from blob_cache where swhid=?", (str(swhid),))
				cache = self.db.fetchone()
				if cache:
				blob = cache[0]
				return blob

				def put_blob(self, swhid: SWHID, blob):
				self.db.execute("insert into blob_cache values (?, ?)", (str(swhid), blob))
				self.conn.commit()

View Options

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_abstract_io

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				import json
				from pathlib import Path
				import sqlite3

				from swh.model.identifiers import CONTENT, SWHID, parse_swhid
				from swh.web.client.client import typify


				class Cache:
				def __init__(self, cache_dir):
				seirlUnsubmitted Done Inline Actions I would like this class to be more pythonic by working as a context manager. For me, the sqlite3 connection should probably happen in an `__enter__()` function, and closing it in an `__exit__()` function. I know you cannot use a with block in the Fuse operation class (a shame...), but at least it keeps the quirky aspect of this solely in the Fuse class. seirl: I would like this class to be more pythonic by working as a context manager. For me, the…
				seirlUnsubmitted Done Inline Actions As discussed, we probably want to have two caches (they can probably both inherit some common behavior from an abstract base if that's helpful). If you do go ahead with two classes, you will be able to use getitem and setitem to enable a more pythonic way of accessing this cache, like `blob_cache[swhid] = X`. seirl: As discussed, we probably want to have two caches (they can probably both inherit some common…
				if cache_dir == ":memory:":
				self.conn = sqlite3.connect(":memory:")
				else:
				cache_path = Path(cache_dir, "cache.sqlite")
				cache_path.parents[0].mkdir(parents=True, exist_ok=True)
				self.conn = sqlite3.connect(cache_path)

				self.db = self.conn.cursor()
				self.db.execute("create table if not exists metadata_cache (swhid, metadata)")
				self.db.execute("create table if not exists blob_cache (swhid, blob)")
				self.conn.commit()
				zackUnsubmitted Done Inline Actions if the sqlite3 module is a well-behaved Python DB module, this could be better written with a contextmanager that commits at the end zack: if the sqlite3 module is a well-behaved Python DB module, this could be better written with a…

				seirlUnsubmitted Done Inline Actions I think it's weird to keep a long-lived cursor in the class, you should probably create a new one whenever you're executing something without the `conn.execute(` shortcuts. I think the class should only hold the connection as its state, not the currently active cursors. seirl:* I think it's weird to keep a long-lived cursor in the class, you should probably create a new…
				def get_metadata(self, swhid: SWHID):
				zackUnsubmitted Done Inline Actions Please add docstring to all public methods, classes and modules. No need to be very detailed for now, oneliners would do for starters. But it's much better to have them since the beginning than having to add all of them later. zack: Please add docstring to all public methods, classes and modules. No need to be very detailed…
				self.db.execute(
				"select metadata from metadata_cache where swhid=?", (str(swhid),)
				)
				cache = self.db.fetchone()
				if cache:
				metadata = json.loads(cache[0])
				return typify(metadata, swhid.object_type)

				def put_metadata(self, swhid: SWHID, metadata):
				zackUnsubmitted Done Inline Actions methods that do not return anything should be explicitly annotated with `-> None`, otherwise the default return value is `Any` (h/t @olasd ) zack: methods that do not return anything should be explicitly annotated with `-> None`, otherwise…
				self.db.execute(
				"insert into metadata_cache values (?, ?)",
				(
				str(swhid),
				json.dumps(
				metadata,
				# Converts the typified JSON to plain str version
				default=lambda x: x.object_id
				if isinstance(x, SWHID)
				else x.__dict__,
				),
				),
				)
				self.conn.commit()

				def get_metadata_swhids(self):
				self.db.execute("select swhid from metadata_cache")
				swhids = self.db.fetchall()
				swhids = [parse_swhid(x[0]) for x in swhids]
				return swhids

				def get_blob(self, swhid: SWHID):
				if swhid.object_type != CONTENT:
				zackUnsubmitted Done Inline Actions We might want to fix this in swh-web-client, making typify optional in return values, as it's really silly to typify there and goes the other way here. Can you file a dedicated task for this? zack: We might want to fix this in swh-web-client, making typify optional in return values, as it's…
				haltodeAuthorUnsubmitted Done Inline Actions Filed in: T2646. haltode: Filed in: T2646.
				raise AttributeError("Cannot retrieve blob from non-content object type")
				seirlUnsubmitted Done Inline Actions While this fix is here temporarily, make it less hard to read by using parentheses, it's not obvious what the "if" applies to here. seirl: While this fix is here temporarily, make it less hard to read by using parentheses, it's not…

				self.db.execute("select blob from blob_cache where swhid=?", (str(swhid),))
				cache = self.db.fetchone()
				if cache:
				blob = cache[0]
				return blob

				def put_blob(self, swhid: SWHID, blob):
				self.db.execute("insert into blob_cache values (?, ?)", (str(swhid), blob))
				self.conn.commit()

View Options

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_authentication

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				import json
				from pathlib import Path
				import sqlite3

				from swh.model.identifiers import CONTENT, SWHID, parse_swhid
				from swh.web.client.client import typify


				class Cache:
				def __init__(self, cache_dir):
				seirlUnsubmitted Done Inline Actions I would like this class to be more pythonic by working as a context manager. For me, the sqlite3 connection should probably happen in an `__enter__()` function, and closing it in an `__exit__()` function. I know you cannot use a with block in the Fuse operation class (a shame...), but at least it keeps the quirky aspect of this solely in the Fuse class. seirl: I would like this class to be more pythonic by working as a context manager. For me, the…
				seirlUnsubmitted Done Inline Actions As discussed, we probably want to have two caches (they can probably both inherit some common behavior from an abstract base if that's helpful). If you do go ahead with two classes, you will be able to use getitem and setitem to enable a more pythonic way of accessing this cache, like `blob_cache[swhid] = X`. seirl: As discussed, we probably want to have two caches (they can probably both inherit some common…
				if cache_dir == ":memory:":
				self.conn = sqlite3.connect(":memory:")
				else:
				cache_path = Path(cache_dir, "cache.sqlite")
				cache_path.parents[0].mkdir(parents=True, exist_ok=True)
				self.conn = sqlite3.connect(cache_path)

				self.db = self.conn.cursor()
				self.db.execute("create table if not exists metadata_cache (swhid, metadata)")
				self.db.execute("create table if not exists blob_cache (swhid, blob)")
				self.conn.commit()
				zackUnsubmitted Done Inline Actions if the sqlite3 module is a well-behaved Python DB module, this could be better written with a contextmanager that commits at the end zack: if the sqlite3 module is a well-behaved Python DB module, this could be better written with a…

				seirlUnsubmitted Done Inline Actions I think it's weird to keep a long-lived cursor in the class, you should probably create a new one whenever you're executing something without the `conn.execute(` shortcuts. I think the class should only hold the connection as its state, not the currently active cursors. seirl:* I think it's weird to keep a long-lived cursor in the class, you should probably create a new…
				def get_metadata(self, swhid: SWHID):
				zackUnsubmitted Done Inline Actions Please add docstring to all public methods, classes and modules. No need to be very detailed for now, oneliners would do for starters. But it's much better to have them since the beginning than having to add all of them later. zack: Please add docstring to all public methods, classes and modules. No need to be very detailed…
				self.db.execute(
				"select metadata from metadata_cache where swhid=?", (str(swhid),)
				)
				cache = self.db.fetchone()
				if cache:
				metadata = json.loads(cache[0])
				return typify(metadata, swhid.object_type)

				def put_metadata(self, swhid: SWHID, metadata):
				zackUnsubmitted Done Inline Actions methods that do not return anything should be explicitly annotated with `-> None`, otherwise the default return value is `Any` (h/t @olasd ) zack: methods that do not return anything should be explicitly annotated with `-> None`, otherwise…
				self.db.execute(
				"insert into metadata_cache values (?, ?)",
				(
				str(swhid),
				json.dumps(
				metadata,
				# Converts the typified JSON to plain str version
				default=lambda x: x.object_id
				if isinstance(x, SWHID)
				else x.__dict__,
				),
				),
				)
				self.conn.commit()

				def get_metadata_swhids(self):
				self.db.execute("select swhid from metadata_cache")
				swhids = self.db.fetchall()
				swhids = [parse_swhid(x[0]) for x in swhids]
				return swhids

				def get_blob(self, swhid: SWHID):
				if swhid.object_type != CONTENT:
				zackUnsubmitted Done Inline Actions We might want to fix this in swh-web-client, making typify optional in return values, as it's really silly to typify there and goes the other way here. Can you file a dedicated task for this? zack: We might want to fix this in swh-web-client, making typify optional in return values, as it's…
				haltodeAuthorUnsubmitted Done Inline Actions Filed in: T2646. haltode: Filed in: T2646.
				raise AttributeError("Cannot retrieve blob from non-content object type")
				seirlUnsubmitted Done Inline Actions While this fix is here temporarily, make it less hard to read by using parentheses, it's not obvious what the "if" applies to here. seirl: While this fix is here temporarily, make it less hard to read by using parentheses, it's not…

				self.db.execute("select blob from blob_cache where swhid=?", (str(swhid),))
				cache = self.db.fetchone()
				if cache:
				blob = cache[0]
				return blob

				def put_blob(self, swhid: SWHID, blob):
				self.db.execute("insert into blob_cache values (?, ?)", (str(swhid), blob))
				self.conn.commit()

View Options

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_bezier

View Options

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_pdf

View Options

swh/loader/package/pubdev/tests/test_pubdev.py

View Options

swh/loader/package/pubdev/tests/test_tasks.py

pubdev: Modify loader after origin URL modification in related lister
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D8355 (id=30161)

Changes applied before test

Patch application report for D8355 (id=30271)

Changes applied before test

Patch application report for D8355 (id=30273)

Changes applied before test

Patch application report for D8355 (id=30299)

Changes applied before test

Revision Contents
Changeset List

Diff 30273

swh/loader/package/pubdev/loader.py

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_Autolinker

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_abstract_io

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_authentication

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_bezier

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_pdf

swh/loader/package/pubdev/tests/test_pubdev.py

swh/loader/package/pubdev/tests/test_tasks.py

pubdev: Modify loader after origin URL modification in related listerClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D8355 (id=30161)

Changes applied before test

Patch application report for D8355 (id=30271)

Changes applied before test

Patch application report for D8355 (id=30273)

Changes applied before test

Patch application report for D8355 (id=30299)

Changes applied before test

Revision ContentsChangeset List

Diff 30273

swh/loader/package/pubdev/loader.py

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_Autolinker

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_abstract_io

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_authentication

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_bezier

swh/loader/package/pubdev/tests/data/https_pub.dev/api_packages_pdf

swh/loader/package/pubdev/tests/test_pubdev.py

swh/loader/package/pubdev/tests/test_tasks.py

pubdev: Modify loader after origin URL modification in related lister
ClosedPublic
Actions

Revision Contents
Changeset List