rebase

and fix the LOGGER.error usage

remove hardcoded log levels

Use ReadOnlyObjStorage and NonIterableObjStorage instead of NotImplementedError

document the build_objstorage() test helper function

remove useless statement

remove mistakenly commited mypy.ini file

rebase

remove a (comment) garbage line

rebase

slight simplification as suggested by vlorentz

respawn jenkins

respawn jenkins

rebase

split the diff in 2

B3 I am not convinced a "synthetic" flag on the Snapshot branch makes sense, or at least I find this name confusing, especially considering we already have a synthetic flag on Revision: it's not synthetic in the sense of it's not object crafted by SWH, it comes from the origin.

Ok I think what puzzle me in this description is the fact the 2 first bullets of the "git loader adaptations" are actually only one point: at the end of a successful loading, store a mapping in the extid table.

forgotten print statement...

In T3627#71819, @vlorentz wrote:

An alternative to annotating synthetic refs: add a "type" or "forge_type" attribute to snapshots.

What's the difference in deployed dependencies versions (staging vs. prod)?

For ENEA I'd llike to test different scenarios for the source objstorage:

just a quick remark about the scheduling of (sub)tasks of this task: IMHO the autoscaling should come last; all the supervision/monitoring/logging related tasks are much more important than the autoscaling.

Better docstrings and kill a few map()

Thx

be a bit more consistent...

In D6410#167121, @douardda wrote:

In D6410#166398, @ardumont wrote:

as @vlorentz pointed out [1], this change should be irrelevant though...

[1] https://github.com/celery/kombu/blob/master/kombu/serialization.py#L369-L372

does not seem to be the proper fix.

FTR, using the celery cli tool directly from a development venv to interact with the celery server running in the docker compose test setup (as described there ) used to work ok, but not any more.

One have to specify the app, like:

celery --app=swh.scheduler.celery_backend.config.app status

[edit] I use celery 5.1.2 in my venv.

Unless I'm mistaken, this error does not appear in sentry any more, right?

In D6410#166398, @ardumont wrote:

as @vlorentz pointed out [1], this change should be irrelevant though...

[1] https://github.com/celery/kombu/blob/master/kombu/serialization.py#L369-L372

looks fine to me

Ok but see my 2 (nitpicky) comments

LGTM thx

See https://forge.softwareheritage.org/source/swh-perfecthash/

allow the pathslicer to be a noop (with an empty slicing)

LGTM

FTR without D6401, the packfile received from GH for the CocoaPods/Specs repo contains 21162 references, 21146 of which are starting with /refs/pull/ and 7126 are ending with /merge (even if those have been explicitly not asked thanks to the filtering in RepoRepresentation.determine_wanted().
When D6401 is applied, we only get the 20-ish references that are not pull request related.

token for the prod will be needed after that as well, thanks

Also there is no real value in keeping 3 revisions: the last 2 revisions actually improve/modify the code from the first revision.

this should be squashed with the previous diff, and still my previous question about .gitignore

As others (and I) said, this must come with actual documentation.
As is, I have hard time understanding how this actually works (even after reading the document in hedgdoc).

indent...

improve entrypoint script to properly handle a SIGTERM

looks ok to me. Just one question, why do you need __future__.annotation?

Oh well...

Ideally this doc would (briefly) describe how bazaar works and how it is different from already supported DVCS, then document chosen the "mapping" of the bzr model into swh (especially mentioning what is lost during this).