Copy of an email I sent on 2021-11-17:

At least loader deposit and npm [1] are fine.

Here is an overview of the fields (+ internal version name + branch name) used by each package loader, after D6616:

Great news: of the 469k corrupt SVN revisions, all but 14 (yes, 14) can be fixed simply by adding 1 microsecond to their timestamp.

After further investigation, I can't find any directory that is in a completely bad order; they are either ordered like git does (by adding a / at the end of dir entries) or by assuming a null byte at the end of dir entries.

analysis on directories (some are also part of the fixable_trivial above, but I don't have the exact number, I lost it in my analysis):

In T3656#72364, @grouss wrote:

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)
checking in the current live swh DAG 2 days ago 98% have one in release or snapshot_branch.
indeed I was surprised because I did'nt have to loop over the revision history.

You might be interested by what @grouss just opened in T3660
(ah scratched that, zack already mentioned it)

In T3660, @grouss has found many more.
Might be for a different reason (the dataset he analyzed is not the live one), but it's worth a comparison.

My script finished running on releases. Result: all 644k releases are recoverable (mostly just missing gpg signatures), except 75k whose origin does not exist anymore.

this commit has lots of one-line "gpgsig" headers instead of a single "gpgsig" header with a multiline value: https://github.com/git-for-windows/git/commit/5f549aa2f78314ac37bbd436c8f80aea4c752e07

Aggregated stats at the moment:

1. "nonce" header is *after* gpgsig
2. double "author" field in the original, and another commit with three "committer"....
3. "mergetag" headers with an extra newline at the end (current versions of the loader strip it, looks like older ones didn't)
4. "author xxx <yyy@gmail.com> <type 'int'> -0200" in original commit (dulwich obviously can't parse this)

1. some negative UTCs don't have the boolean set to true (the only example I have is also missing a space in the same line, so this might be why)
2. "0000" as offset (with no sign), dulwich can't even parse this
3. "+0575" as offset; parsed as "+0615"
4. "\r" in an email address was stripped

1. "author xxx<yyy@googlemail.com> <xxx<yyy@googlemail.com>> 1282956323 +0200" parsed as "author xxx<yyy@googlemail.com> <xxx <yy@googlemail.com>> 1282956323 +0200" (yes there is a fullname in the email)
2. a few commits with this timezone: "--700", parsed as "+0700"
3. Some weird offsets parsed as "+0000": "+1400"/"+051800"/"+1558601"
4. looks like old git loaders tried to decode and re-encode name and/or email, which messed it up when not UTF8

some new ones:

swh:1:dir:880034219b47a123b97161de4e0d3301746cec75 (from https://github.com/kentnl/Dist-Zilla-PluginBundle-Author-KENTNL) has a single entry; we stored it with mode 40000 on that entry, but the original mode is 40755.

We always add "gpgsig" as the last header of revisions, but sometimes it is not. eg. https://github.com/kuehnelth/secure-nyancatd/commit/10011016fc08401b6d3f05c2561a5dd3eb0a2641

Commits ingested before https://forge.softwareheritage.org/rDLDGfb03140e169f38f5ea0d64d81b524b4b967335f7 have an extra newline at the end of the mergetag header

Two other sources of mismatched checksums:

Old versions of Dulwich (eg. 0.16.3, the version in stretch), dropped newlines at the end of the gpgsig header.

Dulwich 0.19.10 (released in january 2019) changed the way they handle signatures on annotated tags, so we silently drop all signatures since we started using it (probably whenever we upgraded loaders to Buster).

In T3246#63559, @douardda wrote:

do we also intent to have a takedown topic on kafka?

also: what about exports we provide on git annex?

do we also intent to have a takedown topic on kafka?

Knobs to adjust the visibility of origins in the archive and in the web API

yes

I _think_ this usecase is solved with the origin_visit_status table (created vs. ongoing vs. completed). @vlorentz?