Page MenuHomeSoftware Heritage
Create Task
Maniphest T3019

Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql
Closed, MigratedEdits Locked
Actions

Description

  1. new column with the hash
  2. fill it (will need a migration with python code)
  3. add unique index
  4. make the new index a primary key?

Revisions and Commits

rDSTO Storage manager
D5029rDSTO2d540b0580cc Add raw_extrinsic_metadata.id column in postgresql.

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT2685 Test update of deposit with metadata-only in integration tests
 Migratedgitlab-migrationT3197 Mirror: fix common issues of a replayer session
 Migratedgitlab-migrationT3201 Mirror: unsupported Unicode escape sequence
 Migratedgitlab-migrationT2344 Build a connector for software deposit via Zenodo/InvenioRDM
 Migratedgitlab-migrationT2538 Add new option for the CLI swh-deposit for the metadata-only deposit
 Migratedgitlab-migrationT1021 SWORD deposit of metadata about an existing SWH object
 Migratedgitlab-migrationT2537 Extend new deposit endpoint to support metadata-only deposits
 Migratedgitlab-migrationT1258 Synthesize release objects for all upstream things that match the concept of a release
 Migratedgitlab-migrationT2059 Generate (swh) releases from all git tags
 Migratedgitlab-migrationT3089 Remove the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT2513 Copy metadata on revisions to the extrinsic metadata storage
 Migratedgitlab-migrationT3128 Improve deposit integration, management and display
 Migratedgitlab-migrationT2540 support the loading of metadata-only deposits in the metadata storage
 Migratedgitlab-migrationT2779 Put information (client, collection and deposit-id) inside metadata for metadata-only deposit
 Migratedgitlab-migrationT2520 Setup dedicated kafka cluster on new rocquencourt hardware
 Migratedgitlab-migrationT2686 Use hashes for all kafka keys
 Migratedgitlab-migrationT2668 Package loaders should write extrinsic metadata on directories instead of revisions/releases
 Migratedgitlab-migrationT2703 Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects
 Migratedgitlab-migrationT3018 Allow querying raw_extrinsic_metadata by hash in swh-storage
 Migratedgitlab-migrationT3022 Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields
 Migratedgitlab-migrationT3019 Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql
 Migratedgitlab-migrationT3074 Migrate all packages away from the old SWHID class
 Migratedgitlab-migrationT3034 generalize usage of SWHID for referencing SWH archive objects

Event Timeline

vlorentz triaged this task as High priority.Feb 2 2021, 1:37 PM
vlorentz created this task.
vlorentz removed subscribers: rdicosmo, moranegg.Feb 2 2021, 1:40 PM
vlorentz renamed this task from Allow querying raw_extrinsic_metadata by hash in swh.storage.postgresql to Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql.Feb 2 2021, 2:20 PM
vlorentz added a parent task: T3022: Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields.
vlorentz added a subtask: T3074: Migrate all packages away from the old SWHID class.Feb 26 2021, 6:57 PM
vlorentz closed subtask T3074: Migrate all packages away from the old SWHID class as Resolved.Mar 5 2021, 12:31 PM
vlorentz added a revision: D5029: Add raw_extrinsic_metadata.id column in postgresql..Mar 5 2021, 3:52 PM
vlorentz added a commit: rDSTO2d540b0580cc: Add raw_extrinsic_metadata.id column in postgresql..Mar 22 2021, 1:55 PM
olasd mentioned this in T2520: Setup dedicated kafka cluster on new rocquencourt hardware.Mar 23 2021, 2:24 PM
olasd closed this task as Resolved.Mar 23 2021, 2:31 PM

After a lot of back and forth, and the release of swh.model v2.3.0 and swh.storage v0.26.0, this is now all done and deployed in staging and production.

We went the following way for the migration:

  • make swh.model write id fields in the journal
  • deploy swh.storage with the new swh.model (so all writes happen with the new model)
  • run swh storage backfill on the raw_extrinsic_metadata topic to fill the journal with objects using the new model
  • (make sure the journal gets compacted to remove old versions of the object, with a combination of topic.retention.ms and having to run the backfill multiple times for it to work on all the real-world data)
  • run swh storage replay on raw_extrinsic_metadata, using a fork of swh.storage that wrote objects to a new table (using the new schema)
  • once the replayer caught up, run some queries to spot check that all the data got properly migrated
  • once validated, stop the workers; stop replayer; deploy new version of swh.storage with new schema, move the new table in place of the old one (and take care of logical replication); then restart the workers
gitlab-migration changed the task status from Resolved to Migrated.Sun, Jan 8, 4:33 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T3074: Migrate all packages away from the old SWHID class from Resolved to Migrated.Sun, Jan 8, 4:33 PM
About · Developer information · Legal