Page MenuHomeSoftware Heritage

ingest antelink s3 contents in swh
Closed, MigratedEdits Locked

Event Timeline

s3 downloaded contents can be injected in swh.
worker01 is currently doing it.

zack renamed this task from Inject s3 contents in swh to ingest antelink s3 contents in swh.Apr 1 2016, 10:12 AM
zack assigned this task to ardumont.
zack added a project: General.

I forgot to update this, sorry.

This is done but for 34 files whose size are > 4,000,000,000 bytes.
List is https://forge.softwareheritage.org/P52

Currently independently running a final check of what data has been properly imported from s3 before removing it from the source.

I'm only looking at databases, not at the object store.

Check done (source file : a s3 listing Guillaume provided me in december). 85670 unique sha1s are missing.

Thanks.
Missing data reinjected.

Latest missing files (24348 items)

Like i said in an email, those match the corrupted files at ~99,5%, the remaining ones are files whose size > 4Gib.
We are good for s3 bucket deletion.

olasd changed the visibility from "All Users" to "Public (No Login Required)".May 13 2016, 5:09 PM