listing done!

Open it back, it's required in the end. Another fix will happen to propose a default
get_current_version implementation in the BaseDb.

This needs to be reverted in waiting for [1] to be resolved.
I'll attend to it tomorrow.

Drop function for f-string, simpler indeed.

I've started back the loader git on that cluster:

Update tests

(Same for scrubber but idk if it's in docker)

Full listing is not finished yet but still there remains origins with exotic starting urls which are not canonicalized.
I'd say the issue lies with the canonicalize swh.core implementation code which only deals with https:// and git:// urls.
So some improvments are needed there.

heads up, ingestion still ongoing with quite some stability in regards to memory consumption.

But, please make sure the migration script are actually runnable though (i'd say with docker and swh db upgrade cli).
I'm not sure our schema and indexes sql files are idempotent enough for the migration script 134 to work with it.
^ Hence the request changes here.

Not sure i get all this but this rather lgtm

But i don't get why you use of assert within the runtime code instead of raising proper exception instead.

Nope, it's fine without this.

Status update, both worker1.staging and worker17 are beyond the failing step of pack
file limit where they usually crash \o/ [1].

much better ;)

I've started a 32g experiment in worker1.staging and 64g in worker17.

8g (pack size limit) was not enough either, it broke on both workers ¯\_(ツ)_/¯.
We have no clue as to what size limit should be done so i'm clearly taking shots in the dark.
I've started a 32g experiment in worker1.staging and 64g in worker17.
We will see.

This was meant to be a draft, but I couldn't find the button to make it so

missing documentation and rationale

Side node, why aren't there tests alongside this module?

mmm, it's already the case so something is off.

Plan:

• P1369: Listing status after first round listing
• Clean up maven github origins listing [1]
• Trigger maven full run [2]
• Wait for listing to finish
• Listing status after new maven lister round of listing
• Ping in mailing list discussion with data!

worker17 is complaining as well but differently somehow.
same version for both though [2].

Ok, expectedly, it does not work as is [1] ;)
Second run then with twice the actual pack file limit [2].

I've triggered a run on worker1.staging [1] and worker17 as is for now.
We'll see for the pack file size limit after that run fails (if it does).

Should be ready to be deployed now.

Awesome! Thanks.

@douardda Any news on how to modify a db template for the tests?

Is the scheduler section in 'swh::deploy::indexer_journal_client::config' still needed ?

Adapt according to discussion (description and test plan updated already)

Old maven behavior results in origins like git://github.com, ... [1]
The new maven lister behavior should now result in canonical github urls http://github.com/user/repo.
Analysis ongoing and report will go after that comment.

couple of questions inline.

I've skimmed through a bit and this does lgtm from afar so far.