as per title: we want to have a first, complete, off-site mirror of our object storage, hosted on Azure
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T239 preserve at least 2 copies of each content object | ||
| Migrated | gitlab-migration | T691 complete object storage mirror on Azure (meta task) | ||
| Unknown Object (Maniphest Task) |
Event Timeline
Comment Actions
We now have a "full" content mirror on azure (the data for each 16th bucket is up to date as of the time the snapshot was taken).
We still need to process the logs for the full injection.
Comment Actions
@olasd recently made a lot of progress on this one.
Can you leave a note about how many objects (roughly) are now on the Azure object storage and how many are missing?
Comment Actions
- The main archive currently synchronously writes all contents to Azure as well as the local storage (the gap is strictly closing)
- all partitions from uffizi have been copied to azure and mass-injected (except for partition 8 which only got partially mass injected)
- after this process, it looks like azure is missing 10% of all objects (excluding partition 8), which are all on banco
- I've started a procedure to copy the missing objects from banco directly. Estimated time to completion ~ 1 month
- The same procedure has been started to copy the missing objects from partition 8 on uffizi. Estimated time to completion ~ 15 days
Comment Actions
After processing the logs of the backfilling process to make sure to redo all the ranges that were interrupted in various database migrations, I'm now confident that this task is complete: we have a full mirror of all contents on Azure, which is kept up to date by the main archive storage backend writing synchronously to it.