stuff related to the mirroring infrastructure, protocol, and tooling used to maintain the Software Heritage mirror network
Details
Details
Description
Wed, May 11
Wed, May 11
credentials created following https://docs.softwareheritage.org/sysadm/mirror-operations/onboard.html#how-to-create-the-objstorage-credentials
vsellier changed the status of T4234: Add snyk credentials for the staging r/o object storage from Open to Work in Progress.
Mar 25 2022
Mar 25 2022
bchauvet raised the priority of T3116: Roll out at least one operational mirror from Normal to High.
Mar 23 2022
Mar 23 2022
bchauvet renamed T3116: Roll out at least one operational mirror from Roll out first operational mirror to Roll out at least one operational mirror.
Feb 7 2022
Feb 7 2022
moranegg added a comment to T1576: document the typical cost(s) of hosting an archive mirror.
Is this page enough?
https://docs.softwareheritage.org/sysadm/mirror-operations/planning.html
Jan 27 2022
Jan 27 2022
douardda closed T3218: The graph replayer generates REQTMOUT Timeout errors, a subtask of T3197: Mirror: fix common issues of a replayer session, as Resolved.
Seems moslty ok, even if these errors still pop now and then in the logs.
Jan 24 2022
Jan 24 2022
douardda triaged T3881: Mirror - handle handling of multiple kafka messages for the same object as High priority.
Jan 19 2022
Jan 19 2022
Jan 18 2022
Jan 18 2022
Jan 4 2022
Jan 4 2022
moranegg triaged T3829: Document mirror - how to create and deploy a mirror from scratch as Normal priority.
May 11 2021
May 11 2021
vlorentz closed T3303: Replaying REMD topics is broken, a subtask of T3197: Mirror: fix common issues of a replayer session, as Resolved.
May 4 2021
May 4 2021
vlorentz added a revision to T3303: Replaying REMD topics is broken: D5670: Stop storing authority/fetcher metadata..
Apr 27 2021
Apr 27 2021
zack added a comment to T1576: document the typical cost(s) of hosting an archive mirror.
docs !
moranegg added a comment to T1576: document the typical cost(s) of hosting an archive mirror.
Where should this information be available?
- on the main website
- docs
- a private location
Apr 23 2021
Apr 23 2021
Apr 22 2021
Apr 22 2021
douardda added a comment to T3200: Mirror: year is out of range.
These revisions are probably coming from https://gitlab.com/gitlab-org/gitlab-test (or a clone)
douardda added a comment to T3200: Mirror: year is out of range.
Ah fun, one of the revisions with this pb, on staging (ba3343bc4fa403a8dfbfcab7fc1a8c29ee34bd69) seems to have been crafted by https://gitlab.com/gitlab-org/gitlab-foss/-/blob/staging-26-fix_add_deploy_key_spec/spec/models/merge_request_diff_commit_spec.rb
Apr 21 2021
Apr 21 2021
douardda added a comment to T3200: Mirror: year is out of range.
See T3170 (error generated by the same invalid kafka messages).
Apr 19 2021
Apr 19 2021
olasd closed T2003: Content replayer may try to copy objects before they are available from an objstorage, a subtask of T1914: Keep mirror of contents on S3 up to date, as Resolved.
Apr 8 2021
Apr 8 2021
douardda added a comment to T3198: Mirror: unexpected closed connection to the pg server.
Just got this one below. Note that this occurred just when the replayer actually started to insert object in the storage (before that, since the start of the replayer process, only kafka scaffolding took place for quite some time, around 30mn!)
Apr 6 2021
Apr 6 2021
douardda closed T3201: Mirror: unsupported Unicode escape sequence as Resolved by committing rDSTO39507b24d0f4: Make the replayer drop the Revision.metadata.
douardda closed T3201: Mirror: unsupported Unicode escape sequence, a subtask of T3197: Mirror: fix common issues of a replayer session, as Resolved.
Apr 2 2021
Apr 2 2021
douardda added a comment to T3197: Mirror: fix common issues of a replayer session.
Currently, the mirror test session is running with:
douardda added a comment to T3201: Mirror: unsupported Unicode escape sequence.
easy fix: modify the replayer to ignore this 'metadata' column while inserting revisions
douardda added a comment to T3201: Mirror: unsupported Unicode escape sequence.
09:45 <+vlorentz> douardda: yes and the only way around it (short of dropping data) is T3089 09:46 -swhbot:#swh-devel- T3089 (submitter: vlorentz, owner: vlorentz, status: Open): Remove the 'metadata' column of the 'revision' table <https://forge.softwareheritage.org/T3089> 09:46 <+vlorentz> or switching to cassandra 09:46 <+vlorentz> the good news is, they couldn't be inserted in the storage either, so you can safely drop them for now
douardda triaged T3199: Mirror: key value violates unique constraint "person_fullname_idx" as High priority.
Mar 15 2021
Mar 15 2021
Mar 11 2021
Mar 11 2021
rdicosmo updated the task description for T3116: Roll out at least one operational mirror.