Page MenuHomeSoftware Heritage

MirrorFolder
ActivePublic

Members

  • This project does not have any members.
  • View All

Watchers

  • This project does not have any watchers.
  • View All

Details

Description

stuff related to the mirroring infrastructure, protocol, and tooling used to maintain the Software Heritage mirror network

Recent Activity

Wed, May 11

vsellier closed T4234: Add snyk credentials for the staging r/o object storage as Resolved.

credentials created following https://docs.softwareheritage.org/sysadm/mirror-operations/onboard.html#how-to-create-the-objstorage-credentials

Wed, May 11, 3:53 PM · Mirror, System administration
vsellier changed the status of T4234: Add snyk credentials for the staging r/o object storage from Open to Work in Progress.
Wed, May 11, 3:40 PM · Mirror, System administration

Mar 25 2022

bchauvet raised the priority of T3116: Roll out at least one operational mirror from Normal to High.
Mar 25 2022, 5:27 PM · Roadmap 2022, Unknown Object (Project), Mirror, Roadmap 2021, meta-task

Mar 23 2022

bchauvet renamed T3116: Roll out at least one operational mirror from Roll out first operational mirror to Roll out at least one operational mirror.
Mar 23 2022, 4:42 PM · Roadmap 2022, Unknown Object (Project), Mirror, Roadmap 2021, meta-task

Feb 7 2022

moranegg added a comment to T1576: document the typical cost(s) of hosting an archive mirror.

Is this page enough?
https://docs.softwareheritage.org/sysadm/mirror-operations/planning.html

Feb 7 2022, 11:58 AM · Documentation, Mirror

Jan 27 2022

douardda closed T3218: The graph replayer generates REQTMOUT Timeout errors, a subtask of T3197: Mirror: fix common issues of a replayer session, as Resolved.
Jan 27 2022, 1:43 PM · Mirror
douardda closed T3218: The graph replayer generates REQTMOUT Timeout errors as Resolved.

Seems moslty ok, even if these errors still pop now and then in the logs.

Jan 27 2022, 1:43 PM · Mirror

Jan 24 2022

douardda triaged T3883: Handle updated kafka messages for the objstorage replayer as High priority.
Jan 24 2022, 5:25 PM · Mirror
douardda triaged T3882: Handle updated kafka messages for the storage replayer as High priority.
Jan 24 2022, 5:22 PM · Storage manager, Mirror
douardda triaged T3881: Mirror - handle handling of multiple kafka messages for the same object as High priority.
Jan 24 2022, 5:22 PM · Mirror

Jan 19 2022

moranegg triaged T3865: Update diagram of mirror architecture as Normal priority.
Jan 19 2022, 4:42 PM · Mirror, Documentation
vsellier added a revision to T3829: Document mirror - how to create and deploy a mirror from scratch: D6979: Document the mirror credentials management.
Jan 19 2022, 3:23 PM · Mirror
olasd added a revision to T3829: Document mirror - how to create and deploy a mirror from scratch: D6978: Update docker stack configuration to match documentation changes.
Jan 19 2022, 2:56 PM · Mirror
ardumont added a revision to T3829: Document mirror - how to create and deploy a mirror from scratch: D6967: docs: Fix redondant extlink link warnings on archive.s.o.
Jan 19 2022, 2:32 PM · Mirror
ardumont added a revision to T3829: Document mirror - how to create and deploy a mirror from scratch: D6965: mirror-operations: Convert markup to restructuredtext.
Jan 19 2022, 2:31 PM · Mirror
ardumont added a revision to T3829: Document mirror - how to create and deploy a mirror from scratch: D6969: Update mirror docker docs following a walkthrough.
Jan 19 2022, 2:30 PM · Mirror

Jan 18 2022

vsellier added a revision to T3829: Document mirror - how to create and deploy a mirror from scratch: D6966: Move mirror documentation from devel to sysadm.
Jan 18 2022, 5:00 PM · Mirror

Jan 4 2022

moranegg triaged T3829: Document mirror - how to create and deploy a mirror from scratch as Normal priority.
Jan 4 2022, 12:15 PM · Mirror

May 11 2021

vlorentz closed T3303: Replaying REMD topics is broken, a subtask of T3197: Mirror: fix common issues of a replayer session, as Resolved.
May 11 2021, 4:23 PM · Mirror
vlorentz closed T3303: Replaying REMD topics is broken as Resolved.
May 11 2021, 4:23 PM · Mirror

May 4 2021

vlorentz added a revision to T3303: Replaying REMD topics is broken: D5670: Stop storing authority/fetcher metadata..
May 4 2021, 4:07 PM · Mirror
douardda triaged T3303: Replaying REMD topics is broken as High priority.
May 4 2021, 10:11 AM · Mirror

Apr 27 2021

zack added a comment to T1576: document the typical cost(s) of hosting an archive mirror.

docs !

Apr 27 2021, 4:06 PM · Documentation, Mirror
moranegg added a comment to T1576: document the typical cost(s) of hosting an archive mirror.

Where should this information be available?

  • on the main website
  • docs
  • a private location
Apr 27 2021, 3:48 PM · Documentation, Mirror

Apr 23 2021

vlorentz assigned T3116: Roll out at least one operational mirror to douardda.
Apr 23 2021, 4:49 PM · Roadmap 2022, Unknown Object (Project), Mirror, Roadmap 2021, meta-task

Apr 22 2021

douardda added a comment to T3200: Mirror: year is out of range.

These revisions are probably coming from https://gitlab.com/gitlab-org/gitlab-test (or a clone)

Apr 22 2021, 12:10 PM · Mirror
douardda added a comment to T3200: Mirror: year is out of range.

Ah fun, one of the revisions with this pb, on staging (ba3343bc4fa403a8dfbfcab7fc1a8c29ee34bd69) seems to have been crafted by https://gitlab.com/gitlab-org/gitlab-foss/-/blob/staging-26-fix_add_deploy_key_spec/spec/models/merge_request_diff_commit_spec.rb

Apr 22 2021, 12:05 PM · Mirror

Apr 21 2021

douardda added a comment to T3200: Mirror: year is out of range.

See T3170 (error generated by the same invalid kafka messages).

Apr 21 2021, 6:58 PM · Mirror

Apr 19 2021

olasd closed T2003: Content replayer may try to copy objects before they are available from an objstorage, a subtask of T1914: Keep mirror of contents on S3 up to date, as Resolved.
Apr 19 2021, 12:06 PM · Mirror, Datasets

Apr 8 2021

douardda added a comment to T3198: Mirror: unexpected closed connection to the pg server.

Just got this one below. Note that this occurred just when the replayer actually started to insert object in the storage (before that, since the start of the replayer process, only kafka scaffolding took place for quite some time, around 30mn!)

Apr 8 2021, 12:02 PM · Mirror
douardda triaged T3218: The graph replayer generates REQTMOUT Timeout errors as High priority.
Apr 8 2021, 11:44 AM · Mirror

Apr 6 2021

douardda closed T3201: Mirror: unsupported Unicode escape sequence as Resolved by committing rDSTO39507b24d0f4: Make the replayer drop the Revision.metadata.
Apr 6 2021, 4:42 PM · Mirror
douardda closed T3201: Mirror: unsupported Unicode escape sequence, a subtask of T3197: Mirror: fix common issues of a replayer session, as Resolved.
Apr 6 2021, 4:42 PM · Mirror
vlorentz added a revision to T3201: Mirror: unsupported Unicode escape sequence: D5414: Make the replayer drop the Revision.metadata.
Apr 6 2021, 4:25 PM · Mirror
vlorentz added a subtask for T3201: Mirror: unsupported Unicode escape sequence: T3089: Remove the 'metadata' column of the 'revision' table.
Apr 6 2021, 2:20 PM · Mirror

Apr 2 2021

douardda added a comment to T3197: Mirror: fix common issues of a replayer session.

Currently, the mirror test session is running with:

Apr 2 2021, 10:15 AM · Mirror
douardda added a comment to T3201: Mirror: unsupported Unicode escape sequence.

easy fix: modify the replayer to ignore this 'metadata' column while inserting revisions

Apr 2 2021, 10:05 AM · Mirror
douardda added a comment to T3201: Mirror: unsupported Unicode escape sequence.
09:45 <+vlorentz> douardda: yes and the only way around it (short of dropping data) is T3089
09:46 -swhbot:#swh-devel- T3089 (submitter: vlorentz, owner: vlorentz, status: Open): Remove the 'metadata' column of the 'revision' table <https://forge.softwareheritage.org/T3089>
09:46 <+vlorentz> or switching to cassandra
09:46 <+vlorentz> the good news is, they couldn't be inserted in the storage either, so you can safely drop them for now
Apr 2 2021, 9:59 AM · Mirror
douardda triaged T3201: Mirror: unsupported Unicode escape sequence as High priority.
Apr 2 2021, 9:54 AM · Mirror
douardda triaged T3200: Mirror: year is out of range as High priority.
Apr 2 2021, 9:51 AM · Mirror
douardda triaged T3199: Mirror: key value violates unique constraint "person_fullname_idx" as High priority.
Apr 2 2021, 9:48 AM · Mirror
douardda triaged T3198: Mirror: unexpected closed connection to the pg server as High priority.
Apr 2 2021, 9:47 AM · Mirror
douardda triaged T3197: Mirror: fix common issues of a replayer session as High priority.
Apr 2 2021, 9:41 AM · Mirror

Mar 15 2021

vlorentz triaged T3116: Roll out at least one operational mirror as Normal priority.
Mar 15 2021, 12:28 PM · Roadmap 2022, Unknown Object (Project), Mirror, Roadmap 2021, meta-task

Mar 11 2021

rdicosmo updated the task description for T3116: Roll out at least one operational mirror.
Mar 11 2021, 7:59 PM · Roadmap 2022, Unknown Object (Project), Mirror, Roadmap 2021, meta-task
rdicosmo merged task T2209: At least 2 full mirrors up and running into T3116: Roll out at least one operational mirror.
Mar 11 2021, 7:57 PM · Mirror, Roadmap 2020
rdicosmo merged T2209: At least 2 full mirrors up and running into T3116: Roll out at least one operational mirror.
Mar 11 2021, 7:57 PM · Roadmap 2022, Unknown Object (Project), Mirror, Roadmap 2021, meta-task
rdicosmo added a parent task for T2914: mirror documentation: add ballpark storage/infra requirements: T3116: Roll out at least one operational mirror.
Mar 11 2021, 7:56 PM · Mirror, Documentation
rdicosmo added a parent task for T1576: document the typical cost(s) of hosting an archive mirror: T3116: Roll out at least one operational mirror.
Mar 11 2021, 7:56 PM · Documentation, Mirror
rdicosmo added subtasks for T3116: Roll out at least one operational mirror: T2914: mirror documentation: add ballpark storage/infra requirements, T1576: document the typical cost(s) of hosting an archive mirror.
Mar 11 2021, 7:56 PM · Roadmap 2022, Unknown Object (Project), Mirror, Roadmap 2021, meta-task