@ardumont These timeouts are either caused by the high CPU usage of swh-storage, or an unrelated issue.

@ardumont These timeouts are either caused by the high CPU usage of swh-storage, or an unrelated issue.

There now remains errors :

• directly related to indexers (bad input) so indexer fails
• Sometimes it's the indexer's api call to the indexer storage that fails. I've created a dashboard to see those errors [1]

To exercise my hypothesis, i've increased those timeouts for storage0.euwest.azure's storage service endpoints:

• revision_get: 500ms to 2s
• directory_ls: 20s to 40s

oh, i think i know... [1] We have a timeout limit on some endpoints... (including the revision_get endpoint mentioned in the stacktrace).

Once in a while, an indexer worker crashes with the following error:

I agree we must be careful with not bloating swh.core, but the current subject, it really makes sense to me to put this basic db access wrapper as a core functionality.

The object storage doesn't have content names, so it cannot address this feature as stated.

Resolved by D645.

Indeed, you're right the timezone offset is used to compute a revision identifier so even if its value is incorrect it should be stored anyway.

The simplest solution would be to check if the computed timezone offset lies in the adequate bounds [UTC−14:00, UTC+14:00] and set it to 0 if not.

objstorage already has an in-memory implementation?

objstorage already has an in-memory implementation?

After a few days of rest:

there is a good chance that this series of diffs break stuffs in the doc generation or some (test) tools at swh-environment level... These will be checked and fixed ASA this series is considered ready to be merged.

\o/