A few remarks:

well this task should be closed, and a new subtask could be added for the alerting

unless I'm mistaken, this task can be closed now, it looks to have reached a steady state where the lag is near 0

I've been thinking a bit about the refactoring of the ProvenanceStorageServer as described in the doc, with a series of queues between the public API and the backend database.

typos

there is a typo in the commit message

overall ok, but I'd like to see the comments about fixtures addressed first.

nice job, thx

I agree some more tests and validations needs to be done on this storage schema, but can we please land it for now as is? I've put a warning in the documentation (in D6031) to point the fact this flavor is not "production ready". cc @aeviso

FTR I've tried to investigate a bit to find clues of what the origin of the outage was, but I did not find any obvious culprit.

ok then

return itertools.chain([res], stream_results(f, page_token = res.page_token, **kwargs))

why not something like:

rebase

rebase

rebase

rebase

move _relation_uses_location_table at the end of the class

fix typos reported by ardumont and vlorentz (thx)

In D5843#155651, @douardda wrote:

In D5843#155564, @aeviso wrote:

It's not clear to me how the denormalized version handles the insertion of duplicated entries.

It's something I am still trying to figure also (whether this code performs as expected under heavy concurrent workload). I want to make more tests (by hand, this is hard to implement as a "unit" test) ASAP.

I am not fond at all of the code duplication (between R-C and O-R synth file parsers), looks to me at least parts of it could be kept factorised in a dedicated module (I agree it should not live in conftest any more: too much code and logic now). It would then be best to have these test-helper functions tested themselves (as unitary as possible).

In D5843#155564, @aeviso wrote:

It's not clear to me how the denormalized version handles the insertion of duplicated entries.

In D5843#155564, @aeviso wrote:

It's not clear to me how the denormalized version handles the insertion of duplicated entries.

rebase and cpitalize sql queries

capitalize sql querie

LGTM but see my questions (not sure they make really sense, but who knows)

In T3444#67730, @ardumont wrote:

ceph is not properly monitored (ENOSPC should not get unoticed on these machines),

P1099 and further earlier logs from that moment do not seem to warn about this... T3945
got created for this.

Potential issues/weakness of our current infra:

I would have loved to also replace the logic in relation_add() and _relation_get() by stored SQL functions, but it's above my poor SQL skills...

rebase

sorry I don't understand everything here:

I still disagree with the implementation of get_dates() but meh

I've made several small comments / nitpicks, fell free to address them or not.

okay but as stated, I don't like too much the general usage of the RealDictCursor; sometimes it helps, but sometimes it does not. Ideally both should be available (depending on the query).

ok but please remove print statements before

In D5943#152508, @vlorentz wrote:

Why do all these queries use LOCK TABLE?

ok but the SQL query could be improved to not return unwanted dates

This initial proposal from CINES has not been selected because it de facto normalize a number of relations of the SWH graph making it unfit to storage in a solution like Vitam (too many objects, hard to manage incremental updates).

1. Proposal from CINES

In D5843#151992, @vlorentz wrote:

Should this be documented somewhere? (How to use it / why)

I think I'd rather like to have an explicit list of excluded fields (when these extra fields are added). So I'd prefer see this diff be something that compares dicts (as a result of BaseObject.to_dict()), possibly filtered to exclude some fields.

Also, at some point we might want to use better templating to write these SQL queries, or use stored procedures (with the proper "variation" being chosen at db creation time on the selected flavor; would simplify the python code a lot.

reword a bit the ci message and kill a few tabs in 30-schema.sql

In D5843#151350, @douardda wrote:

yes I know, names for the subqueries are horrible...

yes I know, names for the subqueries are horrible...

rebase, adapt and implement denormalization for content_in_dir and dir_in_rev

we keep it for now

In D5885#151011, @aeviso wrote:

I believe this diff is duplicated and the other one was already landed.

overall ok but see the comment

rebase