Conclusion, it's mostly [1] ok now. Those who were not usable with the cli tool are now
ok.

• storage [1]
• indexer [2]
• scheduler [3]
• scrubber [4]
• vault: status quo, still not working ¯\_(ツ)_/¯ [5]

Looks like either the loader didn't detect it is a fork, or github sent a large packfile anyway.

In swh/loader/git/loader.py at the end of the prepare function, could you print self.statsd.constant_tags and self.parent_origins, to see which it is?

Loader crashed with memory issues. Probably too much loading in //.
Currently stopping the worker's other processes to let this one finish (i'll restart it).

It works like @vsellier suggested as well.

In D7965#206999, @vsellier wrote:

is it not possible to use a sql script directly, in the idea of [1] ?
I suppose it will make the script simpler as the postgresql image logic will take care of running the script only during the database initialization.

[1] https://forge.softwareheritage.org/source/swh-provenance/browse/master/docker-compose.yml$16

The argument parsing is starting to get unconfortable to read.

One remark from @vsellier got my attention, it will break the docker setup.
Maybe @douardda hinted at it as well...

He is not wrong there, see for example [1].
So some adaptations will be needed there.

[1]

One remark from @vsellier got my attention, it will break the docker setup.
Maybe @douardda hinted at it as well...

Triggered a run to ingest a fork (extra arguments needed with the cli) on production worker:

Success for production worker [1]. Staging worker is still working on it.

Adapt according to exchange

@douardda Any news on how to modify a db template for the tests?

@vlorentz some impact from stopping the concurrency build part ¯\_(ツ)_/¯

There remains git and other dvcs typed origins [1] listed by maven but not github ones [2].

lgtm (probably needs tedious hand-managed-db-migration-in-docker test)

/me closes his eyes and accepts the diff ¯\_(ツ)_/¯

The upgrade was basically migrating files in three clicks but at least I found a cypress regression (https://github.com/cypress-io/cypress/issues/22054) that was fixed really quickly.

Fix current expected code version

/me closes his eyes and accepts the diff ¯\_(ツ)_/¯

Rebase

Update requirements

one remark inline.

status: triggered 2 full-maven lister runs on maven central and jboss [1]
And no more exotic github urls are popping up [2].

Although some stuff are worth keeping, notably the tests on the missing coverage cli (i'll do it in another diff).

Not the proper fix

Without this diff, storage and indexer are actually broken since their datastore is missing the current_version attribute.
But it's simpler to not revert that code (abandon this diff) which somehow clarifies a bit what's searched for and the simple
fix is to add their current_version attribute (the one on db.py becomes unneeded).

Yesterday, i had fixed, diffed, released and pushed the diff [1] fixing the
canonicalization of remaining exotic urls, cleaned up 'git' (out of a maven listing)
origins and triggered back a listing. Today, checking back those origins (staging
scheduler), there was still noise which should no longer have been there...

Reverting:

• Stopping and disabling journal client services [1]
• D7950: Revert puppet manifest changes
• scheduler0.staging: deploy manifest changes [2]
• workers.staging: Deploy manifest changes [3]
• check everything is back to normal [4]