Page MenuHomeSoftware Heritage

Keep mirror of contents on S3 up to date
Closed, MigratedEdits Locked

Description

In view of T1899, this sub-task aims at stopping the gap between the main archive and AWS from growing.

As per discussions from last week I'm assigning this to @olasd, but he still needs input from @seirl on the credentials side to proceed.

Event Timeline

zack renamed this task from synchronously write content objects to AWS to synchronously write content objects to AWS during ingestion.Jul 14 2019, 4:47 PM
zack triaged this task as High priority.
zack created this task.

I don't think we're going to do this but rather use the journal client approach. (Even more so considering that writing to S3 takes 500ms for each object, which sounds like a silly artificial limit to put on a synchronous process).

olasd renamed this task from synchronously write content objects to AWS during ingestion to Keep mirror of contents on S3 up to date.Dec 7 2019, 6:35 PM

We'll need to address T2003 before this can be closed (if we go the journal client route), so marking accordingly.

We've now hit T2003 hard as the client caught up with the head of the local kafka cluster. That's why the curve is flattening out currently, as I stopped the replayers until the queue is implemented.