allow the pathslicer to be a noop (with an empty slicing)

LGTM

FTR without D6401, the packfile received from GH for the CocoaPods/Specs repo contains 21162 references, 21146 of which are starting with /refs/pull/ and 7126 are ending with /merge (even if those have been explicitly not asked thanks to the filtering in RepoRepresentation.determine_wanted().
When D6401 is applied, we only get the 20-ish references that are not pull request related.

token for the prod will be needed after that as well, thanks

Also there is no real value in keeping 3 revisions: the last 2 revisions actually improve/modify the code from the first revision.

this should be squashed with the previous diff, and still my previous question about .gitignore

As others (and I) said, this must come with actual documentation.
As is, I have hard time understanding how this actually works (even after reading the document in hedgdoc).

indent...

improve entrypoint script to properly handle a SIGTERM

looks ok to me. Just one question, why do you need __future__.annotation?

Oh well...

Ideally this doc would (briefly) describe how bazaar works and how it is different from already supported DVCS, then document chosen the "mapping" of the bzr model into swh (especially mentioning what is lost during this).

In T3104#71609, @dachary wrote:

@douardda

SWH I guess: I don't see the difference whether it's embedded in swh-objstorage, winery or a dedicated package.

If I understand correctly, you're suggesting that I create a package at the same level as https://forge.softwareheritage.org/source/puppet-swh-site/, right ? For instance https://forge.softwareheritage.org/source/swh-perfecthash/ by following the instructions from the documentation.

So does it make sense to use this package instead of reimplementing one? What's the catch?

In addition to being unmaintained,

Would it be possible to add a "conception documentation" included in the docs/ of the BZR loader repo? (possibly with D6344 or as a standalone diff)?

In T3104#71460, @dachary wrote:

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

It would but who would maintain it in the long run ?

IMHO This diff should be squashed in D6165 (it's really part of the work adding the rabbitmq-based backend).

as @olasd should be squashed, but meh

Look to me that this open/close interface really should come with a context manager.

I still think it's best to use the wrapped function name as "method" but meh

Looks ok (not sure I really understand the fix however, more precisely, what was the purpose of the revision_start != 1 condition), but I really don't understand the commit message:

In T3104#71408, @douardda wrote:

Ideally, since the perfecthash feature will be needed only for a specific objstorage backend, it should be an optional dependency.

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

Or use this one maybe https://github.com/GregBowyer/cmph-cffi ?

Source for the cmph-cffi package in pypi seems to be https://github.com/venkateshks/cmph-cffi (well at least there are tags in there)

Ideally, since the perfecthash feature will be needed only for a specific objstorage backend, it should be an optional dependency.

add type annotation for process_replay_objects()

rebase

Test coverage looks fairly complete, thx

In T3487#71230, @vsellier wrote:

• postgresql:13
  • 1000 parallel connections allowed

Might be possible to simplify this a bit using a similar approach to https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/metrics.py$16-26

Overall looks ok to me but:

In D6347#164631, @vlorentz wrote:

Why?

Rebase

Better commit message

typos

In D6346#164633, @vlorentz wrote:

I don't see why (I'm guessing for simplification), but ok

Looks fine to me, but it needs some extensive tests indeed.

use types-psycopg2 instead of ignore it in mymy.ini

fine for me

lgtm

thx a lot

LGTM

In D6316#163800, @douardda wrote:

In D6316#163794, @douardda wrote:

You may use fcntl.flock for this

I mean using an empty (lock) file in the opam_root directory.

In D6316#163794, @douardda wrote:

You may use fcntl.flock for this

In D6316#163766, @ardumont wrote:

Following what i said in the loader diff, i'm actually closing this.
Ack on the lock folder but i won't attend to it immediately.

[1] D6318

As i was wrong in my implementation of the loader implementation and @aleo made me realize, i've fixed it.
So now that lister diff becomes relevant again, so claimed it back.

I think there was already a problem before, but since we have now more chance to hit it, I'd really like the opam_init process to lock the directory when running opam commands.

It's a great idea but i've no idea how to actually do that though.

Maybe adding --safe flag [1] during the command that actually list the packages would be enough instead.
I've actually added that for the loader [2] (for the command that also read information)

[1]

--safe, --readonly
    Make sure nothing will be automatically updated or rewritten. Useful for calling from completion scripts, for example. Will fail whenever such an operation is needed ; also avoids waiting for locks, skips interactive
    questions and overrides the $OPAMDEBUG variable. This is equivalent to set environment variable $OPAMSAFE.

[2] D6318

LGTM (not checked everything is accurate nor there are obvious missing services, but it's a huge improvement as is, thx)

In T1805#45984, @vlorentz wrote:

Items 5, 6, 7 aka pagination, auth and batches - I believe these come naturally with item 4 (specification wise)

They don't. OpenAPI is a specification to describe APIs, and it contains absolutely nothing about pagination or batches.

I think there was already a problem before, but since we have now more chance to hit it, I'd really like the opam_init process to lock the directory when running opam commands.

It would be nice to have a README fil in swh/lister/maven/tests/data explaining what the data files are, where they come from, how they have been generated, etc.