Page MenuHomeSoftware Heritage

handle the uffizi content store being full
Closed, MigratedEdits Locked

Description

The uffizi content storage is getting dangerously close to full.

After discussing our requirements (redundancy, migration to ceph, archiver issues), the following plan is being put into place:

  1. reduce archiver lag and improve redundancy : Synchronously write all objects to azure as well as local storage
  2. reduce local storage requirements : Stripe the local storage of contents across uffizi and banco
  3. prepare the ceph migration, improve redundancy : Synchronously write a copy of the objects in ceph, with the final redundancy requirements (erasure coding, 5 chunks + 2 parity chunks), but with a more risky failure domain (OSD instead of host).
  4. reclaim storage space : remove already backed up objects from uffizi and banco
  5. future-proof the system : adapt the archiver to handle striped storages

Event Timeline

olasd changed the task status from Open to Work in Progress.May 3 2018, 4:40 PM
olasd triaged this task as High priority.
olasd created this task.

This was, in fact, solved by adding more storage.

gitlab-migration changed the status of subtask T1045: Implement a striping object storage from Resolved to Migrated.
gitlab-migration changed the status of subtask T1046: Stripe local contents between uffizi and banco from Resolved to Migrated.
gitlab-migration changed the status of subtask T1047: Write all contents synchronously to the ceph cluster from Wontfix to Migrated.