Page MenuHomeSoftware Heritage
Create Task
Maniphest T3197

Mirror: fix common issues of a replayer session
Closed, MigratedEdits Locked
Actions

Description

When running a replay session of the staging area storage, there a few common issues that occur, these should be fixed before starting a real-life mirror session.

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT3197 Mirror: fix common issues of a replayer session
 Migratedgitlab-migrationT3198 Mirror: unexpected closed connection to the pg server
 Migratedgitlab-migrationT3199 Mirror: key value violates unique constraint "person_fullname_idx"
 Migratedgitlab-migrationT3200 Mirror: year is out of range
 Migratedgitlab-migrationT3201 Mirror: unsupported Unicode escape sequence
 Migratedgitlab-migrationT3089 Remove the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT3090 Make loaders not rely on the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT2849 Design and implement a mapping from "original VCS ids" to SWHIDs to help incremental loaders
 Migratedgitlab-migrationT3138 Make loaders stop writing in the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT3139 Make loaders stop reading from the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT3140 Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata
 Migratedgitlab-migrationT3143 Migrate revision metadata to extid in the storage
 Migratedgitlab-migrationT3142 Make loaders write to the ExtId storage
 Migratedgitlab-migrationT3141 Make loaders pass ExtId objects between functions instead of "known_artifact"
 Migratedgitlab-migrationT2564 migrate existing revisions metadata extra_headers to actual extra_headers field
 Migratedgitlab-migrationT2513 Copy metadata on revisions to the extrinsic metadata storage
 Migratedgitlab-migrationT2512 Make all loaders write their extrinsic metadata to the appropriate storage
 Migratedgitlab-migrationT2496 Write deposit metadata for revisions in the generic metadata storage
 Migratedgitlab-migrationT2306 Generic storage for extrinsic, qualified metadata related to any node of the swh archive
 Migratedgitlab-migrationT2703 Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects
 Migratedgitlab-migrationT3017 Use hashes as keys in swh.journal.objects.raw_extrinsic_metadata
 Migratedgitlab-migrationT3018 Allow querying raw_extrinsic_metadata by hash in swh-storage
 Migratedgitlab-migrationT3022 Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields
 Migratedgitlab-migrationT3019 Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql
 Migratedgitlab-migrationT3020 Add an "index" for raw_extrinsic_metadata.id in swh.storage.cassandra
 Migratedgitlab-migrationT3074 Migrate all packages away from the old SWHID class
 Migratedgitlab-migrationT3034 generalize usage of SWHID for referencing SWH archive objects
 Migratedgitlab-migrationT3218 The graph replayer generates REQTMOUT Timeout errors
 Migratedgitlab-migrationT3303 Replaying REMD topics is broken

Event Timeline

douardda triaged this task as High priority.Apr 2 2021, 9:41 AM
douardda created this task.
douardda added a comment.Apr 2 2021, 10:15 AM

Currently, the mirror test session is running with:

ii  python3-swh.core        0.12.0-1~swh1~bpo10+1 all          Software Heritage core utilities
ii  python3-swh.journal     0.7.1-1~swh1~bpo10+1  all          Software Heritage Journal utilities
ii  python3-swh.model       2.3.0-1~swh1~bpo10+1  all          Software Heritage data model
ii  python3-swh.objstorage  0.2.2-1~swh1~bpo10+1  all          Software Heritage Object Storage
ii  python3-swh.scheduler   0.10.0-1~swh1~bpo10+1 all          Software Heritage Scheduler
ii  python3-swh.storage     0.27.0-1~swh1~bpo10+1 all          Software Heritage storage utilities

Using unprivileged access to the journal.

The number of error after ~20h is:

$ docker service logs swh_storage  2>&1 | grep " ERROR:"  | sort | uniq -c
     20 swh_storage.1.yjtfpr7ambpm@libra    | ERROR:root:duplicate key value violates unique constraint "person_fullname_idx"
      1 swh_storage.1.yjtfpr7ambpm@libra    | ERROR:root:server closed the connection unexpectedly
   1679 swh_storage.1.yjtfpr7ambpm@libra    | ERROR:root:unsupported Unicode escape sequence
   1087 swh_storage.1.yjtfpr7ambpm@libra    | ERROR:root:year 292278994 is out of range
douardda closed subtask T3201: Mirror: unsupported Unicode escape sequence as Resolved.Apr 6 2021, 4:42 PM
vlorentz closed subtask T3303: Replaying REMD topics is broken as Resolved.May 11 2021, 4:23 PM
douardda closed subtask T3218: The graph replayer generates REQTMOUT Timeout errors as Resolved.Jan 27 2022, 1:43 PM
gitlab-migration changed the status of subtask T3201: Mirror: unsupported Unicode escape sequence from Resolved to Migrated.Jan 8 2023, 10:02 PM
gitlab-migration changed the status of subtask T3218: The graph replayer generates REQTMOUT Timeout errors from Resolved to Migrated.
gitlab-migration changed the status of subtask T3303: Replaying REMD topics is broken from Resolved to Migrated.
gitlab-migration closed this task as Migrated.Jan 8 2023, 10:22 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration closed subtask T3198: Mirror: unexpected closed connection to the pg server as Migrated.Jan 8 2023, 10:22 PM
gitlab-migration closed subtask T3199: Mirror: key value violates unique constraint "person_fullname_idx" as Migrated.
gitlab-migration closed subtask T3200: Mirror: year is out of range as Migrated.
About · Developer information · Legal