So what about exports of the archive available on git-annex?

In T3246#63559, @douardda wrote:

do we also intent to have a takedown topic on kafka?

Note that there is the same transient vs cumulative discrepency on the "Accepted requests" graph.

In T1481#63785, @douardda wrote:

I think the "submitted requests per visit type / status" graph should be split in 2 parts. Both accepted and rejected are cumulative values that will indefinitely grow, while pending are transient value aiming at staying near zero, so it makes no sense to have them on the same graph.

I think the "submitted requests per visit type / status" graph should be split in 2 parts. Both accepted and rejected are cumulative values that will indefinitely grow, while pending are transient value aiming at staying near zero, so it makes no sense to have them on the same graph.

In T3084#63569, @douardda wrote:

is there a grafana dashboard dedicated to this queue?

In D5553#140866, @anlambert wrote:

Looks good to me. It's true the black version we are using is quite outdated.

We should upgrade black in all other swh repositories in the same manner then.

rebas

rebase

rebased

rebased

The loading tasks created during this first listing were oneshot tasks. So they have been modified to recurring tasks with something like:

Doesn't this deserve a state-of-the-art kind of thing? Are there documentation material on the subject? How does other (big) cassandra users handle this?

The listing task has been disabled, I think because of failures in the last executions:

is there a grafana dashboard dedicated to this queue?

also: what about exports we provide on git annex?

do we also intent to have a takedown topic on kafka?

Just got this one below. Note that this occurred just when the replayer actually started to insert object in the storage (before that, since the start of the replayer process, only kafka scaffolding took place for quite some time, around 30mn!)

looks like there is no revision with date or committer_date > 9999-12-31 in the main storage...

fix commit message

In D5413#137649, @douardda wrote:

In D5413#137363, @vlorentz wrote:

Could you add a test for the storage? All other *_add have a journal test IIRC

let me check that

rebased

Add explicit checks for extid being written in the journal and split the revision in 2

In D5413#137363, @vlorentz wrote:

Could you add a test for the storage? All other *_add have a journal test IIRC

In D5413#137370, @ardumont wrote:

lgtm

(I would have made that 2 commits with each its own perimeter, 1 for the actual perimeter, 1 to refactor the test, but whatever)

Currently, the mirror test session is running with:

easy fix: modify the replayer to ignore this 'metadata' column while inserting revisions

In D5387#136812, @vlorentz wrote:

test coverage of the code touched by this diff isn't great

In D5389#136825, @vlorentz wrote:

Why .hex() everywhere? Does swh-provenance use hex strings internally?

looks indeed reasonable (both the 1. point and the code) thanks

rebase

rebase

rebased

apply vlorentz comments

refactor a bit the test