I've grown tired of babysitting this, so I've added systemd notify calls to the journal replayer, allowing us to just use the systemd watchdog to restart hung processes.

In T1954#39027, @zack wrote:

So, the amount of contents on S3 went up fairly quickly during in between Nov 10th and Nov 20th, but then it stopped again, is it expected/normal?
(thanks for the metric, it really helps)

So, the amount of contents on S3 went up fairly quickly during in between Nov 10th and Nov 20th, but then it stopped again, is it expected/normal?

I've added a metric with the S3 objects to https://grafana.softwareheritage.org/d/jScG7g6mk/objstorage-object-counts. There's... "some" work to do still.

So I've deployed this (by hand for now) on uffizi and it seems to be doing its job.

I didn't know objects are compressed. That indeed makes the issue harder.

Same here. Not that much an easy hack. And what is the real life use case that drive this feature request? YAGNI?

The object storage doesn't have content names, so it cannot address this feature as stated.

The object storage doesn't have content names, so it cannot address this feature as stated.

A crude script doing this: https://forge.softwareheritage.org/source/snippets/browse/master/vlorentz/get_metadata_files_examples.py

I suppose it all depends on the current storage's configuration.

objstorage_pathslicing manipulates a *gzipped* file object, which means that TTBOMK seek is not supported, and we will have to decompress the complete beginning of the file to get to the range that we really want to read.

In the objstorage's pathslicing implementation, there is the get_stream implementation which is not used [1]

objstorage already has an in-memory implementation?

objstorage already has an in-memory implementation?