The thing is, *everything* can be done using the extrinsic metadata storage instead of the main storage. But that means the data becomes less structured / schemaless, and I don't think that's a good thing.

Build is green

fix compat with python <3.9

At that rate, I'm starting to wonder why we even bother with release/revision objects for deposits at all. If all substantial operations to understand their context need to go through the extrinsic metadata attached to the directory, we could just make the snapshot point to the directory directly?

Cool, nice pytest trick to know !

Build has FAILED

make the test less hacky

LGTM now. :)

Sure! But I'd rather do it later -> T3704

Build is green

actually fetch the metadata from opam, instead of just the first line...

I would rather keep it explicit, it's too easy to overlook it otherwise.

assert_last_visit_matches and check_snapshot should probably be merged somehow:

Maybe it would be worth making the second example the default implementation for this function?

Looks good to me.

Current status:
Following the last discussions, the current track I'm trying to implement is to create a grafana dashboard displaying the current status of the infrastructure.
To do so, some information managed by grafana should be displayed like the end-to-end checks status.

After some tweaks[1], the git bare cooking is finally working correctly on g5k.

Build is green

move to _hash_cffi swh.perfecthash._hash_cffi

In T3627#73323, @zack wrote:

Thanks for the summaries @olasd, both here and on list.
I've followed up on list.

Meanwhile here's what I propose we do (spoiler!):

a) A4: add to the archive Merkle DAG only the filtered snapshot (referencing "intrinsic" branches only, as per A2) and its transitive closure

Build is green

Update: Address @vlorentz comments

2M with unique sha1 index (Embedded arrays with bulk write on content and directory)
(Exited after 542,000 of memory errors)

Build is green

Update: Simplify test implementation

Any idea why this happens?

Good point :) I forgot to update the service urls.

Postgres: 10k with sha1 index (Embedded arrays with bulk write on content and directory)
python client.py -n 4 -C config.yml

will extract a truncated version of that file if it contains a null byte

Postgres: 10k with sha1 index (Embedded arrays without bulk write)
python client.py -n 4 -C config.yml

Where is the documentation on how to access the new read-only object storage?
(hint hint :-))

T3702 and T3703 were created for the basic auth and keycloak authentication support

I still need to do a pass through the C code, which "feels" very dense. Some of it has debug statements which help navigate, but a lot doesn't so it's not too easy to go through (but that's mostly because it's been a while since I've been serious about reading some C code :D).

Build is green

Update: Improve commit messages in test

The issue is related to the difference of fetched revision data whether we use the swh.loader.svn.SvnLoader class (fetching revisions
one at a time through a ping pong with the svn server) or the swh.loader.svn.SvnLoaderFromRemoteDump class (fetching all revisions
to a dump file in one operation).

Original comment posted in T3695 by @ardumont