rebas

rebase

rebased

rebased

The loading tasks created during this first listing were oneshot tasks. So they have been modified to recurring tasks with something like:

Doesn't this deserve a state-of-the-art kind of thing? Are there documentation material on the subject? How does other (big) cassandra users handle this?

The listing task has been disabled, I think because of failures in the last executions:

is there a grafana dashboard dedicated to this queue?

also: what about exports we provide on git annex?

do we also intent to have a takedown topic on kafka?

Just got this one below. Note that this occurred just when the replayer actually started to insert object in the storage (before that, since the start of the replayer process, only kafka scaffolding took place for quite some time, around 30mn!)

looks like there is no revision with date or committer_date > 9999-12-31 in the main storage...

fix commit message

In D5413#137649, @douardda wrote:

In D5413#137363, @vlorentz wrote:

Could you add a test for the storage? All other *_add have a journal test IIRC

let me check that

rebased

Add explicit checks for extid being written in the journal and split the revision in 2

In D5413#137363, @vlorentz wrote:

Could you add a test for the storage? All other *_add have a journal test IIRC

In D5413#137370, @ardumont wrote:

lgtm

(I would have made that 2 commits with each its own perimeter, 1 for the actual perimeter, 1 to refactor the test, but whatever)

Currently, the mirror test session is running with:

easy fix: modify the replayer to ignore this 'metadata' column while inserting revisions

In D5387#136812, @vlorentz wrote:

test coverage of the code touched by this diff isn't great

In D5389#136825, @vlorentz wrote:

Why .hex() everywhere? Does swh-provenance use hex strings internally?

looks indeed reasonable (both the 1. point and the code) thanks

rebase

rebase

rebased

apply vlorentz comments

refactor a bit the test

Add forgotten test data file CMDBTS.msgpack

apply aeviso comments and reorder (and fix) revisions

Rebase and use test dataset from the CMDBTS git repo

rebase and remove the dependency on pytz

LGTM, but I would have loved to see a test dedicated to branches (and filters) with non-ascii chars (ach time I see a <str>.encode() I expect the worst... )

Can you add a line in the commit message explaining what this swh_authority_url config entry is for? Because il looks weird to add use a real URL as value in a docker test environment...

Otherwise LGTM. I'd really like a better commit message, and probably some documentation somewhere (in docs/ maybe?) explaining these 2 levels of metadata, especially documenting the second layer, since it's crafted by the deposit.