Page MenuHomeSoftware Heritage

handle the uffizi content store being full
Started, Work in Progress, HighPublic

Description

The uffizi content storage is getting dangerously close to full.

After discussing our requirements (redundancy, migration to ceph, archiver issues), the following plan is being put into place:

  1. reduce archiver lag and improve redundancy : Synchronously write all objects to azure as well as local storage
  2. reduce local storage requirements : Stripe the local storage of contents across uffizi and banco
  3. prepare the ceph migration, improve redundancy : Synchronously write a copy of the objects in ceph, with the final redundancy requirements (erasure coding, 5 chunks + 2 parity chunks), but with a more risky failure domain (OSD instead of host).
  4. reclaim storage space : remove already backed up objects from uffizi and banco
  5. future-proof the system : adapt the archiver to handle striped storages

Event Timeline

olasd created this task.May 3 2018, 4:40 PM
olasd changed the task status from Open to Work in Progress.
olasd triaged this task as High priority.
olasd changed the status of subtask T1047: Write all contents synchronously to the ceph cluster from Open to Work in Progress.