Hum. This diff is a mess after the refactoring of the main loop.
Please give me some time to fix all of the current issues, then rebase the diff to make it more readable. Hope it will clear up the lines mix.

Build is green

• maven-lister: Fix various comments from review (review D6133).
• maven-lister: add README.md to describe design decisions (review D6133).
• maven-lister: Fix useless storage of file + minor fixes (review D6133).

Build is green

rebased patch

Build is green

Refactor and make the install/update idempotent

whitespace fix

Update

In D6324#163917, @olasd wrote:

Is the broken_dates hunk supposed to be part of this diff? Looks out of place.

Looks good to me, just a small indentation issue to fix before landing this. Thanks !

I'd suggest letting this one stew for a bit for others to have a chance to object :-)

Is the broken_dates hunk supposed to be part of this diff? Looks out of place.

Build is green

Build is green

shouldn't have staged this change

include changes suggested by anlambert which I missed

In D6271#163819, @ardumont wrote:

oh i missed that, great ;)

Complete proposal for the above solution:

oh i missed that, great ;)

Complete proposal to implement the above solution:

Adapt according to review

In D6165#163784, @vlorentz wrote:

What do you mean by assigning requests randomly?

The way gunicorn assigns requests to worker processes in the existing RPC server

How do you guarantee conflict resolution that way?

What conflict resolution? Sorry I didn't really follow swh-provenance's development, and I can't find this in the documentation

Approved, but please fix the minutia I'm mentioning in the above comment before landing.

In D6316#163800, @douardda wrote:

In D6316#163794, @douardda wrote:

You may use fcntl.flock for this

I mean using an empty (lock) file in the opam_root directory.

Actually use the right date!

In D6316#163794, @douardda wrote:

You may use fcntl.flock for this

In D6316#163766, @ardumont wrote:

Following what i said in the loader diff, i'm actually closing this.
Ack on the lock folder but i won't attend to it immediately.

[1] D6318

As i was wrong in my implementation of the loader implementation and @aleo made me realize, i've fixed it.
So now that lister diff becomes relevant again, so claimed it back.

I think there was already a problem before, but since we have now more chance to hit it, I'd really like the opam_init process to lock the directory when running opam commands.

It's a great idea but i've no idea how to actually do that though.

Maybe adding --safe flag [1] during the command that actually list the packages would be enough instead.
I've actually added that for the loader [2] (for the command that also read information)

[1]

--safe, --readonly
    Make sure nothing will be automatically updated or rewritten. Useful for calling from completion scripts, for example. Will fail whenever such an operation is needed ; also avoids waiting for locks, skips interactive
    questions and overrides the $OPAMDEBUG variable. This is equivalent to set environment variable $OPAMSAFE.

[2] D6318

What do you mean by assigning requests randomly?

Status on this:

• Added the foss heptapod instance for listing [1]
• Ensure the run went smoothly [2]
• Stop 12 some workers, keep only 4 for the instance is not harassed [3]
• Trigger those new origins for ingestion
• when done (or almost), starts back the other workers [4]
• Update the archive logs when ingestion is done

On second thought, this is a bad idea. Let's solve the bottleneck by computing identifiers directly using model classes instead of having to dictify them.

Build is green

Update docstring

Build is green

Align commit message with diffs