rebase

move _relation_uses_location_table at the end of the class

fix typos reported by ardumont and vlorentz (thx)

In D5843#155651, @douardda wrote:

In D5843#155564, @aeviso wrote:

It's not clear to me how the denormalized version handles the insertion of duplicated entries.

It's something I am still trying to figure also (whether this code performs as expected under heavy concurrent workload). I want to make more tests (by hand, this is hard to implement as a "unit" test) ASAP.

I am not fond at all of the code duplication (between R-C and O-R synth file parsers), looks to me at least parts of it could be kept factorised in a dedicated module (I agree it should not live in conftest any more: too much code and logic now). It would then be best to have these test-helper functions tested themselves (as unitary as possible).

In D5843#155564, @aeviso wrote:

It's not clear to me how the denormalized version handles the insertion of duplicated entries.

In D5843#155564, @aeviso wrote:

It's not clear to me how the denormalized version handles the insertion of duplicated entries.

rebase and cpitalize sql queries

capitalize sql querie

LGTM but see my questions (not sure they make really sense, but who knows)

In T3444#67730, @ardumont wrote:

ceph is not properly monitored (ENOSPC should not get unoticed on these machines),

P1099 and further earlier logs from that moment do not seem to warn about this... T3945
got created for this.

Potential issues/weakness of our current infra:

I would have loved to also replace the logic in relation_add() and _relation_get() by stored SQL functions, but it's above my poor SQL skills...

rebase

sorry I don't understand everything here:

I still disagree with the implementation of get_dates() but meh

I've made several small comments / nitpicks, fell free to address them or not.

okay but as stated, I don't like too much the general usage of the RealDictCursor; sometimes it helps, but sometimes it does not. Ideally both should be available (depending on the query).

ok but please remove print statements before

In D5943#152508, @vlorentz wrote:

Why do all these queries use LOCK TABLE?

ok but the SQL query could be improved to not return unwanted dates

This initial proposal from CINES has not been selected because it de facto normalize a number of relations of the SWH graph making it unfit to storage in a solution like Vitam (too many objects, hard to manage incremental updates).

1. Proposal from CINES

In D5843#151992, @vlorentz wrote:

Should this be documented somewhere? (How to use it / why)

I think I'd rather like to have an explicit list of excluded fields (when these extra fields are added). So I'd prefer see this diff be something that compares dicts (as a result of BaseObject.to_dict()), possibly filtered to exclude some fields.

Also, at some point we might want to use better templating to write these SQL queries, or use stored procedures (with the proper "variation" being chosen at db creation time on the selected flavor; would simplify the python code a lot.

reword a bit the ci message and kill a few tabs in 30-schema.sql

In D5843#151350, @douardda wrote:

yes I know, names for the subqueries are horrible...

yes I know, names for the subqueries are horrible...

rebase, adapt and implement denormalization for content_in_dir and dir_in_rev

we keep it for now

In D5885#151011, @aeviso wrote:

I believe this diff is duplicated and the other one was already landed.

overall ok but see the comment

rebase

rebase

typos

ok but some questions/remarks have not been addressed...

Thanks for the DatetimeCache & co.

I agree having access to the logs of the task (more or less) in real-time would be very handy (as one can expect on any CI-like tool nowadays).

but please add a comment in the ArchivePostgreSQL's version of snapshot_get_heads explaining why it's (for now) a duplication of the other implementation, thanks

In D5862#150645, @aeviso wrote:

OK for the first two items, but I don't agree on the third one. The idea is to replace one of them by a direct SQL query in the near future so reworking this will be useless. I just didn't implement the query because I needed to move forward with the other stuff

thanks, the diff looks much simpler now :-)

I know I am rambling, but could it come with some testing?

use 'branches' instead of "revisions" as section in the yaml file

Mostly nitpicking comments, but I'd really prefer that:

• the cache is kept properly typed
• the cache clearing thing gets its own git revision

Add a revision to fix the annotation of InMemory's value_sanitizer

small simplification in the docstring

use get_journal_writer in test_stream

fix docstring, thx ardumont

Thanks for the split. Almost there, but a few nitpicks/comments remains:

Thanks a lot for the split.

update the copyright in __init__.py