Page MenuHomeSoftware Heritage
Create Task
Maniphest T1914

Keep mirror of contents on S3 up to date
Closed, MigratedEdits Locked
Actions

Description

In view of T1899, this sub-task aims at stopping the gap between the main archive and AWS from growing.

As per discussions from last week I'm assigning this to @olasd, but he still needs input from @seirl on the credentials side to proceed.

Related Objects
Search...

Event Timeline

zack renamed this task from synchronously write content objects to AWS to synchronously write content objects to AWS during ingestion.Jul 14 2019, 4:47 PM
zack triaged this task as High priority.
zack created this task.
olasd added a comment.Dec 7 2019, 6:32 PM

I don't think we're going to do this but rather use the journal client approach. (Even more so considering that writing to S3 takes 500ms for each object, which sounds like a silly artificial limit to put on a synchronous process).

olasd renamed this task from synchronously write content objects to AWS during ingestion to Keep mirror of contents on S3 up to date.Dec 7 2019, 6:35 PM
olasd added a subtask: T2003: Content replayer may try to copy objects before they are available from an objstorage.

We'll need to address T2003 before this can be closed (if we go the journal client route), so marking accordingly.

douardda mentioned this in T2003: Content replayer may try to copy objects before they are available from an objstorage.Jan 23 2020, 1:53 PM
olasd added a comment.Jan 23 2020, 2:17 PM

We've now hit T2003 hard as the client caught up with the head of the local kafka cluster. That's why the curve is flattening out currently, as I stopped the replayers until the queue is implemented.

rdicosmo closed this task as a duplicate of T1954: Up-to-date objstorage mirror on S3.Mar 4 2021, 5:44 PM
olasd closed subtask T2003: Content replayer may try to copy objects before they are available from an objstorage as Resolved.Apr 19 2021, 12:05 PM
gitlab-migration changed the task status from Duplicate to Migrated.Jan 8 2023, 4:28 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T2003: Content replayer may try to copy objects before they are available from an objstorage from Resolved to Migrated.Jan 8 2023, 4:28 PM
About · Developer information · Legal