just a quick remark about the scheduling of (sub)tasks of this task: IMHO the autoscaling should come last; all the supervision/monitoring/logging related tasks are much more important than the autoscaling.

Better docstrings and kill a few map()

Thx

be a bit more consistent...

In D6410#167121, @douardda wrote:

In D6410#166398, @ardumont wrote:

as @vlorentz pointed out [1], this change should be irrelevant though...

[1] https://github.com/celery/kombu/blob/master/kombu/serialization.py#L369-L372

does not seem to be the proper fix.

FTR, using the celery cli tool directly from a development venv to interact with the celery server running in the docker compose test setup (as described there ) used to work ok, but not any more.

One have to specify the app, like:

celery --app=swh.scheduler.celery_backend.config.app status

[edit] I use celery 5.1.2 in my venv.

Unless I'm mistaken, this error does not appear in sentry any more, right?

In D6410#166398, @ardumont wrote:

as @vlorentz pointed out [1], this change should be irrelevant though...

[1] https://github.com/celery/kombu/blob/master/kombu/serialization.py#L369-L372

looks fine to me

Ok but see my 2 (nitpicky) comments

LGTM thx

[FTR] See https://forge.softwareheritage.org/source/swh-perfecthash/

allow the pathslicer to be a noop (with an empty slicing)

LGTM

FTR without D6401, the packfile received from GH for the CocoaPods/Specs repo contains 21162 references, 21146 of which are starting with /refs/pull/ and 7126 are ending with /merge (even if those have been explicitly not asked thanks to the filtering in RepoRepresentation.determine_wanted().
When D6401 is applied, we only get the 20-ish references that are not pull request related.

token for the prod will be needed after that as well, thanks

Also there is no real value in keeping 3 revisions: the last 2 revisions actually improve/modify the code from the first revision.

this should be squashed with the previous diff, and still my previous question about .gitignore

As others (and I) said, this must come with actual documentation.
As is, I have hard time understanding how this actually works (even after reading the document in hedgdoc).

indent...

improve entrypoint script to properly handle a SIGTERM

looks ok to me. Just one question, why do you need __future__.annotation?

Oh well...

Ideally this doc would (briefly) describe how bazaar works and how it is different from already supported DVCS, then document chosen the "mapping" of the bzr model into swh (especially mentioning what is lost during this).

In T3104#71609, @dachary wrote:

@douardda

SWH I guess: I don't see the difference whether it's embedded in swh-objstorage, winery or a dedicated package.

If I understand correctly, you're suggesting that I create a package at the same level as https://forge.softwareheritage.org/source/puppet-swh-site/, right ? For instance https://forge.softwareheritage.org/source/swh-perfecthash/ by following the instructions from the documentation.

So does it make sense to use this package instead of reimplementing one? What's the catch?

In addition to being unmaintained,

Would it be possible to add a "conception documentation" included in the docs/ of the BZR loader repo? (possibly with D6344 or as a standalone diff)?

In T3104#71460, @dachary wrote:

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

It would but who would maintain it in the long run ?

IMHO This diff should be squashed in D6165 (it's really part of the work adding the rabbitmq-based backend).

as @olasd should be squashed, but meh

Look to me that this open/close interface really should come with a context manager.

I still think it's best to use the wrapped function name as "method" but meh

Looks ok (not sure I really understand the fix however, more precisely, what was the purpose of the revision_start != 1 condition), but I really don't understand the commit message:

In T3104#71408, @douardda wrote:

Ideally, since the perfecthash feature will be needed only for a specific objstorage backend, it should be an optional dependency.

Wouldn't it make sense to put the cffi-based cmph wrapper in a dedicated python module/project (not necessarily under the swh namespace)?

Or use this one maybe https://github.com/GregBowyer/cmph-cffi ?

Source for the cmph-cffi package in pypi seems to be https://github.com/venkateshks/cmph-cffi (well at least there are tags in there)

Ideally, since the perfecthash feature will be needed only for a specific objstorage backend, it should be an optional dependency.

add type annotation for process_replay_objects()

rebase

Test coverage looks fairly complete, thx

In T3487#71230, @vsellier wrote:

• postgresql:13
  • 1000 parallel connections allowed

Might be possible to simplify this a bit using a similar approach to https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/metrics.py$16-26

Overall looks ok to me but:

In D6347#164631, @vlorentz wrote:

Why?

Rebase

Better commit message

typos

In D6346#164633, @vlorentz wrote:

I don't see why (I'm guessing for simplification), but ok

Looks fine to me, but it needs some extensive tests indeed.

use types-psycopg2 instead of ignore it in mymy.ini