Your diff sounds quite enough.

Or even directly the archive loader's default behavior (append previously seen branch
from early snapshot/visit). As discussed, I'm wondering whether an archive loader (gnu
or cran [1]) would not benefit from always displaying previously seen branches (whether
they are still present in the current main api we list/visit or not [2]).

This could be implemented by adding a new option to the loader.

Now deployed in prod:

Neat :)

New swh.loader.core deployed in staging.

if you remember the crash times (.zsh_history?), we could find a range of candidate SWHIDs...

The migration script has now run to completion (took around a week).

I've deployed the extid schema changes on all storages, and I've started the migration script on getty.

https://forge.softwareheritage.org/source/snippets/browse/master/vlorentz/migrate_extid.py

now it is. (the mercurial loader technically doesn't use ExtID internally, but it already passes nodeids, which are close enough)

actually, only solved for package loaders

Build is green

Rework commit message (aligns with diff)

In D5071#128386, @ardumont wrote:

I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader

I don't think anything would break. I'm just not sure the "indirection" would be clear in terms of code readability...

I was wrong. mypy is not happy.

Some signatures would need changing, notably the load, prepare, prepare_origin_visit to drop the spurious (i think) {*args, **kwargs} we are declaring.
(I don't think they are still used any more now, aside for the extra logging arguments)

But still, i'd be confortable if we go that way to do it in another diff (maybe).

In D5071#128386, @ardumont wrote:

I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader

I don't think anything would break. I'm just not sure the "indirection" would be clear in terms of code readability...

I was wrong. mypy is not happy.

Some signatures would need changing, notably the load, prepare, prepare_origin_visit to drop the spurious (i think) {*args, **kwargs} we are declaring.
(I don't think they are still used any more now, aside for the extra logging arguments)

But still, i'd be confortable if we go that way to do it in another diff (maybe).

$ tox -e mypy
GLOB sdist-make: /home/tony/work/inria/repo/swh/swh-environment/swh-loader-core/setup.py
...
mypy run-test: commands[0] | mypy swh
swh/loader/package/loader.py:289: error: Signature of "load" incompatible with supertype "BaseLoader"
swh/loader/package/loader.py:381: error: Argument "date" to "OriginVisit" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/loader.py:570: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/loader.py:692: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/loader.py:727: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/loader.py:756: error: Argument "discovery_date" to "RawExtrinsicMetadata" has incompatible type "Optional[datetime]"; expected "datetime"
swh/loader/package/pypi/loader.py:151: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/npm/loader.py:177: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/nixguix/loader.py:218: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/deposit/loader.py:208: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/deposit/loader.py:244: error: Signature of "load" incompatible with supertype "BaseLoader"
swh/loader/package/debian/loader.py:237: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/cran/loader.py:129: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
swh/loader/package/archive/loader.py:170: error: Item "None" of "Optional[datetime]" has no attribute "isoformat"
Found 14 errors in 8 files (checked 71 source files)

I wonder what would break if the new methods were just put in BaseLoader and the PackageLoader was made to inherit BaseLoader

Build is green