It's currently running on the main storage implementation belvedere.
It needs to be deployed on all storage instances independently on how storage instances are populated (wal replication, ...).
As a consequence, scrubber must be deployed to scrub somerset as well.
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T3135 Improve integrity of ingested content | ||
| Migrated | gitlab-migration | T3841 regularly scrub all the data stores of swh | ||
| Migrated | gitlab-migration | T4371 Deploy swh-scrubber on all storage instances |
Deployed both in staging and production [1]:
15:48:20 swh-scrubber@belvedere:5432=> select now(), datastore, count(*) from corrupt_object group by datastore; +-------------------------------+-----------+-------+ | now | datastore | count | +-------------------------------+-----------+-------+ | 2022-08-04 13:48:26.124778+00 | 4997 | 4482 | | 2022-08-04 13:48:26.124778+00 | 1 | 90899 | +-------------------------------+-----------+-------+ (2 rows) Time: 152.881 ms 15:48:26 swh-scrubber@belvedere:5432=> select now(), * from datastore; +-------------------------------+------+---------+------------+-------------------------------------------------------------------------------------------------------+ | now | id | package | class | instance | +-------------------------------+------+---------+------------+-------------------------------------------------------------------------------------------------------+ | 2022-08-04 13:48:35.116671+00 | 1 | storage | postgresql | user=guest password=xxx dbname=softwareheritage host=db.internal.softwareheritage.org port=5432 | | 2022-08-04 13:48:35.116671+00 | 4997 | storage | postgresql | user=guest password=xxx dbname=softwareheritage host=somerset.internal.softwareheritage.org port=5432 | +-------------------------------+------+---------+------------+-------------------------------------------------------------------------------------------------------+ (2 rows) Time: 69.960 ms