Page MenuHomeSoftware Heritage

Core LoaderFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

loader plumbings used by other loaders

Recent Activity

Wed, Feb 17

anlambert added a revision to T3030: Improve loaders to deal with new visit status events: D5096: common/origin_save: Handle new "not_found" origin visit status.
Wed, Feb 17, 4:45 PM · Core Loader
ardumont closed D5071: Unify loader instantiation.
Wed, Feb 17, 12:03 PM · Core Loader

Tue, Feb 16

swh-public-ci added a comment to D5071: Unify loader instantiation.

Build is green

Tue, Feb 16, 7:11 PM · Core Loader
ardumont updated the diff for D5071: Unify loader instantiation.

Rework commit message (aligns with diff)

Tue, Feb 16, 7:09 PM · Core Loader
olasd accepted D5071: Unify loader instantiation.

I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader

I don't think anything would break. I'm just not sure the "indirection" would be clear in terms of code readability...

I was wrong. mypy is not happy.

Some signatures would need changing, notably the load, prepare, prepare_origin_visit to drop the spurious (i think) {*args, **kwargs} we are declaring.
(I don't think they are still used any more now, aside for the extra logging arguments)

But still, i'd be confortable if we go that way to do it in another diff (maybe).

Tue, Feb 16, 6:57 PM · Core Loader
ardumont updated the summary of D5071: Unify loader instantiation.
Tue, Feb 16, 6:57 PM · Core Loader
ardumont added a comment to D5071: Unify loader instantiation.

I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader

I don't think anything would break. I'm just not sure the "indirection" would be clear in terms of code readability...

I was wrong. mypy is not happy.

Some signatures would need changing, notably the load, prepare, prepare_origin_visit to drop the spurious (i think) {*args, **kwargs} we are declaring.
(I don't think they are still used any more now, aside for the extra logging arguments)

But still, i'd be confortable if we go that way to do it in another diff (maybe).

$ tox -e mypy
GLOB sdist-make: /home/tony/work/inria/repo/swh/swh-environment/swh-loader-core/setup.py
...
mypy run-test: commands[0] | mypy swh
swh/loader/package/loader.py:289: error: Signature of "load" incompatible with supertype "BaseLoader"
swh/loader/package/loader.py:381: error: Argument "date" to "OriginVisit" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/loader.py:570: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/loader.py:692: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/loader.py:727: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/loader.py:756: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/pypi/loader.py:151: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/npm/loader.py:177: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/nixguix/loader.py:218: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/deposit/loader.py:208: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/deposit/loader.py:244: error: Signature of "load" incompatible with supertype "BaseLoader"
swh/loader/package/debian/loader.py:237: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/cran/loader.py:129: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/archive/loader.py:170: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
Found 14 errors in 8 files (checked 71 source files)
Tue, Feb 16, 6:52 PM · Core Loader
ardumont added a comment to D5071: Unify loader instantiation.

I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader

Tue, Feb 16, 6:33 PM · Core Loader
swh-public-ci added a comment to D5071: Unify loader instantiation.

Build is green

Tue, Feb 16, 6:23 PM · Core Loader
ardumont updated the diff for D5071: Unify loader instantiation.

Adapt according to review:

  • Drop swh.loader.pattern and move class Loader in swh.loader.core.loader module
  • Drop unneeded self.create_authorities, self.create_fetchers
Tue, Feb 16, 6:21 PM · Core Loader
ardumont added a comment to D5071: Unify loader instantiation.

@ardumont points out that the base PackageLoader doesn't inherit from BaseLoader, which explains the new (common) base class. I think the new class could just as well be next to BaseLoader, and doesn't warrant the introduction of a pattern module.

Tue, Feb 16, 6:20 PM · Core Loader
olasd added a comment to D5071: Unify loader instantiation.

@ardumont points out that the base PackageLoader doesn't inherit from BaseLoader, which explains the new (common) base class. I think the new class could just as well be next to BaseLoader, and doesn't warrant the introduction of a pattern module.

Tue, Feb 16, 6:07 PM · Core Loader
ardumont added a comment to D5071: Unify loader instantiation.

Are the new pattern module / pattern.Loader class really needed? It looks like these methods could live in the BaseLoader class directly.

Tue, Feb 16, 5:35 PM · Core Loader
olasd added a comment to D5071: Unify loader instantiation.

This is great, thanks!

Tue, Feb 16, 5:33 PM · Core Loader
swh-public-ci added a comment to D5071: Unify loader instantiation.

Build is green

Tue, Feb 16, 10:20 AM · Core Loader
ardumont updated the diff for D5071: Unify loader instantiation.

Add missing test on cli run edge case

Tue, Feb 16, 10:18 AM · Core Loader
swh-public-ci added a comment to D5071: Unify loader instantiation.

Build is green

Tue, Feb 16, 10:00 AM · Core Loader
ardumont updated the diff for D5071: Unify loader instantiation.

Fix unused import

Tue, Feb 16, 9:58 AM · Core Loader
Harbormaster failed remote builds in B19231: Diff 18127 for D5071: Unify loader instantiation!
Tue, Feb 16, 9:54 AM · Core Loader
swh-public-ci added a comment to D5071: Unify loader instantiation.

Build has FAILED

Tue, Feb 16, 9:54 AM · Core Loader
ardumont updated the diff for D5071: Unify loader instantiation.
  • Add missing test on Deposit.from_configfile
  • Drop unneeded conditional in cli
Tue, Feb 16, 9:53 AM · Core Loader

Mon, Feb 15

ardumont updated the summary of D5071: Unify loader instantiation.
Mon, Feb 15, 6:12 PM · Core Loader
vlorentz accepted D5071: Unify loader instantiation.
Mon, Feb 15, 4:43 PM · Core Loader

Fri, Feb 12

ardumont updated the summary of D5071: Unify loader instantiation.
Fri, Feb 12, 8:17 PM · Core Loader
ardumont updated the summary of D5071: Unify loader instantiation.
Fri, Feb 12, 8:12 PM · Core Loader
ardumont updated the summary of D5071: Unify loader instantiation.
Fri, Feb 12, 5:51 PM · Core Loader
ardumont updated the summary of D5071: Unify loader instantiation.
Fri, Feb 12, 5:08 PM · Core Loader

Tue, Feb 9

vsellier added a revision to T3030: Improve loaders to deal with new visit status events: D5052: loader: mark visit as 'not_found' when relevant.
Tue, Feb 9, 5:09 PM · Core Loader
vsellier added a revision to T3030: Improve loaders to deal with new visit status events: D5049: loader: Mark visit status as failed when relevant.
Tue, Feb 9, 3:57 PM · Core Loader
ardumont added a revision to T3030: Improve loaders to deal with new visit status events: D5046: loader.svn: Mark visit status as not_found when relevant.
Tue, Feb 9, 11:58 AM · Core Loader

Mon, Feb 8

ardumont added a revision to T3030: Improve loaders to deal with new visit status events: D5042: loader.svn: Explicit the failure test cases.
Mon, Feb 8, 7:07 PM · Core Loader
ardumont added a revision to T3030: Improve loaders to deal with new visit status events: D5041: loader.git: Mark visit status as not_found when relevant.
Mon, Feb 8, 5:51 PM · Core Loader
ardumont added a revision to T3030: Improve loaders to deal with new visit status events: D5040: loader.git: Explicit the failure test cases.
Mon, Feb 8, 5:50 PM · Core Loader
ardumont updated the task description for T3030: Improve loaders to deal with new visit status events.
Mon, Feb 8, 5:06 PM · Core Loader
ardumont added a revision to T3030: Improve loaders to deal with new visit status events: D5039: core.loader: Allow vcs loaders to deal with not_found status.
Mon, Feb 8, 5:02 PM · Core Loader

Fri, Feb 5

ardumont added a revision to T3030: Improve loaders to deal with new visit status events: D5035: package: Mark visit as not_found when relevant.
Fri, Feb 5, 5:49 PM · Core Loader
ardumont renamed T3030: Improve loaders to deal with new visit status events from Improve loader to deal with new visit status events to Improve loaders to deal with new visit status events.
Fri, Feb 5, 3:40 PM · Core Loader
zack added a project to T3030: Improve loaders to deal with new visit status events: Core Loader.
Fri, Feb 5, 3:40 PM · Core Loader

Nov 20 2020

ardumont closed T2537: Extend new deposit endpoint to support metadata-only deposits, a subtask of T1021: SWORD deposit of metadata about an existing SWH object, as Resolved.
Nov 20 2020, 10:20 AM · Core Loader, SWORD deposit

Nov 18 2020

ardumont added a subtask for T1021: SWORD deposit of metadata about an existing SWH object: T2537: Extend new deposit endpoint to support metadata-only deposits.
Nov 18 2020, 11:33 AM · Core Loader, SWORD deposit

Nov 3 2020

ardumont moved T871: Migrate swh-storage api functions relative to indexers to swh-indexer from Backlog to Archived on the SWORD deposit board.
Nov 3 2020, 4:07 PM · SWORD deposit, Core Loader, Web app, Development environment, Storage manager, Indexer
ardumont moved T872: Deploy and restart indexers from Backlog to Archived on the SWORD deposit board.
Nov 3 2020, 4:07 PM · SWORD deposit, Core Loader, Storage manager, Web app, Puppet recipes, Indexer

Oct 26 2020

douardda added projects to T2585: Properly handle deleted/no longer available origins: Core Loader, Lister.
Oct 26 2020, 2:41 PM · Lister, Core Loader

Sep 25 2020

ardumont closed D4012: core.loader: Log information about origin currently being ingested.
Sep 25 2020, 9:04 AM · Core Loader
ardumont added a comment to D4012: core.loader: Log information about origin currently being ingested.

Our logging handler swh.core.logger.JournalHandler already knows how to pull some metadata from the celery tasks:
...

Sep 25 2020, 9:02 AM · Core Loader
olasd accepted D4012: core.loader: Log information about origin currently being ingested.
Sep 25 2020, 9:00 AM · Core Loader
swh-public-ci added a comment to D4012: core.loader: Log information about origin currently being ingested.

Build is green

Sep 25 2020, 8:58 AM · Core Loader
ardumont added inline comments to D4012: core.loader: Log information about origin currently being ingested.
Sep 25 2020, 8:56 AM · Core Loader
ardumont updated the diff for D4012: core.loader: Log information about origin currently being ingested.

Adapt according to suggestion

Sep 25 2020, 8:56 AM · Core Loader

Sep 24 2020

olasd added a comment to D4012: core.loader: Log information about origin currently being ingested.

To be clear, my main issue today, when I try to look through our logs to
investigate or plain read what's going on (after a deployment for example), I
don't have any clues immediately...

In my mind, the kibana information is not enough by itself, so i think i need
to cross information with say sentry to have some more context... It's
currently quite frustrating... up to an eventual point of, "oh well, I have
some other urgent matters somewhere else..." (sometimes I push through but
sometimes, I fail).

Sep 24 2020, 4:59 PM · Core Loader