Unfortunately, the operator test is a failure due to the lack of configuration possibility

• non blocker, the init containers are OOMkilled during the start, it can be solved by editing the cassandra statefulset created by the operator to extend the limits
• blocker, it's not possible to configure the commitlog_directory explicitly. it's by default on /var/lib/cassandra/commitlog
  • it's not easy to propagate the host mounts to use 2 mountpoints /srv/cassandra and /srv/cassandra/commitlog without tweaking the kernel / rancher configuration
  • it's not possible to add a second volume on the pod description created by the operator

Finally, the cluster is up.
I'm not sure what unstuck the node registration, but I suspect a node with all the roles is needed to bootstrap the cluster.
I tried this initially, it didn't worked, but I'm not sure in which status the cluster was.

The management nodes were correctly created but it seems rancher is having some issuer to register them in the cluster.

the Git loader now exports a swh_loader_filtered_objects_total metric. We should generalize this to other loaders eventually; using one of the options above

Feature has been implemented and deployed, closing this.

done ^

New fix needed so another round for v1.3.1

Thanks olasd for restarting the service following this documentation: https://docs.gunicorn.org/en/stable/signals.html#upgrading-to-a-new-binary-on-the-fly

First, replace the old binary with a new one, then send a USR2 signal to the current master process. It executes a new binary whose PID file is postfixed with .2 (e.g. /var/run/gunicorn.pid.2), which in turn starts a new master process and new worker processes:
At this point, two instances of Gunicorn are running, handling the incoming requests together. To phase the old instance out, you have to send a WINCH signal to the old master process, and its worker processes will start to gracefully shut down.

Sorry the description was not completely clear
The durations metrics are still updated, but not the operations count ones
In 2 metrics at 10 minutes of interval:

vsellier@saam ~ % ls -al /tmp/m[12]              
-rw-r--r-- 1 vsellier vsellier 176850 Mar 30 07:25 /tmp/m1
-rw-r--r-- 1 vsellier vsellier 176846 Mar 30 07:34 /tmp/m2

swh.storage 1.2.0 increased a bunch of timeouts and made some queries smarter so it's entirely plausible that the number of errors has dropped drastically.

What exact metric are you saying isn't updating? In your diff, the swh_storage_request_duration_seconds_count{endpoint="index"} metric seems to be increasing normally

SGTM, thanks!

I have the feeling that, in terms of API extensibility, we'll want to be returning both the OriginVisit and its latest OriginVisitStatus.
If we don't do that, we might find ourselves in a situation where we want to combine the fields from both objects into one, which feels a bit clunky when we could just return a proper composite type.

I have the feeling that, in terms of API extensibility, we'll want to be returning both the OriginVisit and its latest OriginVisitStatus.

swh-model 5.0.0 released, which finalizes these changes

Closing this, because we already fixed everything we can for now.

In T3841#80779, @olasd wrote:

I think it's fine to remove the entries when we don't need them anymore (i.e. the object has been restored). Worst case, it'll be re-added at the next iteration of the script :-)

In T3841#80779, @olasd wrote:

You'll need a column for which datastore has the corrupted object.

You'll need a column for which datastore has the corrupted object.

I wrote a script to scrub postgres and kafka: https://forge.softwareheritage.org/source/snippets/browse/master/vlorentz/recheck_consistency.py