Page MenuHomeSoftware Heritage

Keep mirror of contents on S3 up to date
Open, HighPublic

Description

In view of T1899, this sub-task aims at stopping the gap between the main archive and AWS from growing.

As per discussions from last week I'm assigning this to @olasd, but he still needs input from @seirl on the credentials side to proceed.

Event Timeline

zack renamed this task from synchronously write content objects to AWS to synchronously write content objects to AWS during ingestion.Jul 14 2019, 4:47 PM
zack triaged this task as High priority.
zack created this task.
olasd added a comment.Sat, Dec 7, 6:32 PM

I don't think we're going to do this but rather use the journal client approach. (Even more so considering that writing to S3 takes 500ms for each object, which sounds like a silly artificial limit to put on a synchronous process).

olasd renamed this task from synchronously write content objects to AWS during ingestion to Keep mirror of contents on S3 up to date.Sat, Dec 7, 6:35 PM

We'll need to address T2003 before this can be closed (if we go the journal client route), so marking accordingly.