In T3656#72364, @grouss wrote:

according to the list of nodes provided by seirl there were ~21,000,000 revisions without ancestors according to swh-graph snapshot (2020-12-15)

In T3660, @grouss has found many more.
Might be for a different reason (the dataset he analyzed is not the live one), but it's worth a comparison.

+1 on dropping the / -> /devel/ redirect and have at / a landing page allowing to choose between the 3 bodies of documentation.

Aside from the specific needs of the mirroring stack, the question at hand is whether the read-only object storage should be by default open to the public or not.

This should stay pending until we resolve the archiving policy discussion in T3627, so I'm marking it as such.

Thanks for your feedback @olasd. I see three main arguments raised there: (1) the raciness of archiving those data via other means (= related forks), (2) the completeness of our canvassing of synthetic refs, (3) annotating rather than not archiving "synthetic" refs.

In T3627#71640, @ardumont wrote:

No, the snippet mentioned filters out refs whose name starts with refs/pulls and finishes with /merge
(i realize i made a typo in the main description..., it's now fixed)

According to the snippet referenced by @ardumont, all branch names starting with refs/pull/ should be filtered out.
But in the recent snapshot of torvalds/linux there are a lot of branch names like that.
How come?

Approved, but please fix the minutia I'm mentioning in the above comment before landing.

In T1805#70880, @douardda wrote:

• pagination comes from proper usage of links https://swagger.io/docs/specification/links/
• batches comes from proper usage of parameter serialization https://swagger.io/docs/specification/serialization/

it's true these do not come "for free" but I still have the impression there is an "Open API way" of handling these and we should stick to them.

In T3544#69746, @olasd wrote:

I can see a few alternatives to using git:// over tcp:

• Give our swh bot accounts SSH keys, and use that to clone from GitHub over ssh.

Here's an opinionated and prioritized list.

quick comment on the "Miscellaneous" category:

• it's not a great name, and it really feels they are "less important" than the others even if we say explicitly they aren't (or maybe because we say so :-))
• and shouldn't the two items in there (nix, guix) go under "regular crawling" anyway? (that would trivially solve the previous point)

LGTM, thanks! But please note that remaining "-a/--add" reference in the new docstring, which should be changed to "-e/..." for consistency. Please fix that before landing this change.

Nice! And I also like the refactoring out of client.py.

Thanks for this, and for the screenshots, they look gorgeous!

I'm approving this, but please fix the remaining occurrence of 1000 before landing, as per comment above.

looks good!

I'm requesting some minor changes (+ some other changes to be submitted in a separate diff which I've noticed only now, sorry!).

awesome \o/
(diff accepted)

I'm accepting this diff, but note that I've added a few suggestions for improved language above. Please integrate them before this is final.

Wonderful, thanks for adding the order tests! LGTM.

thanks @KShivendu, this is a great start!

@zack
It is okay with you if I add it in the next diff? This one has become extremely long because of lots of build failures.

In D5990#154613, @zack wrote:

Can we have some documentation of the query language, included in this diff?
E.g., a file under docs/ which will then be rendered on docs.s.o as user documentation for how to use the query language.

Can we have some documentation of the query language, included in this diff?
E.g., a file under docs/ which will then be rendered on docs.s.o as user documentation for how to use the query language.

Thanks for this update, great work!

Looks great! I've noted down only some nits.

Please note down on the sides two remaining TODOs about all this:

• adding a test case with a deduplicated source tree, to make sure nodes that are deduplicated at the Merkle DAG level are present multiple times in the output
• adding a test case for a path that is not decodable in utf-8, to make sure it can be handled propertly

Thanks, both the general structure and implementation look OK.
I'm requesting changes to address two main issues:

only minor changes requested to the docstring on my part

In D5952#152586, @ardumont wrote:

I don't really have an ETA yet [1]. We are roughly 67% done for git and 84.6% for svn
[2]. For mercurial, it's not started as other blocking points are being worked on.
Bazaar and cvs origins are listed but we don't have any loader on that front yet.

Thanks a lot for this!

(3) should be ideally implemented in a way that guarantees that extid that were resolvable in previous versions of the mapping will always be resolvable in future versions

I don't understand. Option 3 is to remove relations between extids and SWHID, so it won't be resolvable anymore.

I've the feeling that option (1) will lead in the long run to an explosion on the size of the mapping which will make us eventually converge (slowly) toward option (3).