Page MenuHomeSoftware Heritage

Move the raw imported data off uffizi/banco /srv/storage/space, which is getting full
Started, Work in Progress, HighPublic

Description

(some) loaders store their raw imported data in a subdirectory of /srv/storage/space on uffizi.

Some initial imports have also been archived on that space, as well as on banco.

This allows us to replay loading of an origin with the exact original data if the conversion is somehow buggy (which happened a lot in the early days of the git loader).

This storage is not critical, and does not need to be available on a live partition of our main storage server. We should move it away to make space for more relevant live data.

Event Timeline

olasd created this task.Nov 20 2018, 12:04 PM
olasd triaged this task as High priority.
olasd renamed this task from Move the raw imported data off uffizi /srv/storage/space, which is getting full to Move the raw imported data off uffizi/banco /srv/storage/space, which is getting full.Jan 4 2019, 2:22 PM
olasd updated the task description. (Show Details)
olasd changed the task status from Open to Work in Progress.Jan 4 2019, 2:25 PM

The one-shot task of moving our raw data off uffizi and banco has been started:

  • Created a new azure storage account 'archiveeuwestswh', with "cold" storage semantics/billing
  • started moving the whole contents of uffizi:/srv/storage/space/mirrors/code.google.com
  • started moving the tarred repositories present in banco:/srv/storage/space/mirrors/github.com
olasd added a comment.Jan 10 2019, 1:37 PM

All files from code.google.com have been moved to the archive.

I'll start copying gitorious data.