Page MenuHomeSoftware Heritage
Create Task
Maniphest T3019

Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql
Closed, ResolvedPublic
Actions

Description

  1. new column with the hash
  2. fill it (will need a migration with python code)
  3. add unique index
  4. make the new index a primary key?

Revisions and Commits

rDSTO Storage manager
D5029rDSTO2d540b0580cc Add raw_extrinsic_metadata.id column in postgresql.

Related Objects
Search...

StatusAssignedTask
 ResolvedardumontT2685 Test update of deposit with metadata-only in integration tests
 OpenNoneT3197 Mirror: fix common issues of a replayer session
 ResolveddouarddaT3201 Mirror: unsupported Unicode escape sequence
 ResolvedvlorentzT1021 SWORD deposit of metadata about an existing SWH object
 OpenvlorentzT1258 Synthesize release objects for all upstream things that match the concept of a release
 OpenNoneT2059 Generate (swh) releases from all git tags
 OpenvlorentzT3089 Remove the 'metadata' column of the 'revision' table
 Work in ProgressolasdT2513 Copy metadata on revisions to the extrinsic metadata storage
 Work in ProgressmoraneggT3128 Improve deposit integration, management and display
 OpenNoneT2344 Build a connector for software deposit via Zenodo/InvenioRDM
 OpenNoneT2329 Extend CodeMeta vocabulary to qualify author relationships
 ResolvedmoraneggT1732 Extend metadata for portals depositing software through SWORD
 ResolvedvlorentzT2306 Generic storage for extrinsic, qualified metadata related to any node of the swh archive
 ResolvedmoraneggT2480 Write specification for a Zenodo deposit to SWH
 ResolvedNoneT2538 Add new option for the CLI swh-deposit for the metadata-only deposit
 ResolvedardumontT2537 Extend new deposit endpoint to support metadata-only deposits
 ResolvedvlorentzT2540 support the loading of metadata-only deposits in the metadata storage
 ResolvedvlorentzT2779 Put information (client, collection and deposit-id) inside metadata for metadata-only deposit
 ResolvedNoneT2937 Provide credentials to InvenioRDM for staging
 OpenNoneT3549 Analyze deposit-client on InvenioRDM compatibility
ResolvedolasdT2520 Setup dedicated kafka cluster on new rocquencourt hardware
 OpenolasdT2686 Use hashes for all kafka keys
 ResolvedvlorentzT2668 Package loaders should write extrinsic metadata on directories instead of revisions/releases
 ResolvedvlorentzT2703 Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects
 ResolvedvlorentzT3018 Allow querying raw_extrinsic_metadata by hash in swh-storage
 ResolvedvlorentzT3022 Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields
 ResolvedvlorentzT3019 Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql
 ResolvedvlorentzT3074 Migrate all packages away from the old SWHID class
 OpenNoneT3034 generalize usage of SWHID for referencing SWH archive objects

Event Timeline

vlorentz triaged this task as High priority.Feb 2 2021, 1:37 PM
vlorentz created this task.
vlorentz removed subscribers: rdicosmo, moranegg.Feb 2 2021, 1:40 PM
vlorentz renamed this task from Allow querying raw_extrinsic_metadata by hash in swh.storage.postgresql to Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql.Feb 2 2021, 2:20 PM
vlorentz added a parent task: T3022: Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields.
vlorentz added a subtask: T3074: Migrate all packages away from the old SWHID class.Feb 26 2021, 6:57 PM
vlorentz closed subtask T3074: Migrate all packages away from the old SWHID class as Resolved.Mar 5 2021, 12:31 PM
vlorentz added a revision: D5029: Add raw_extrinsic_metadata.id column in postgresql..Mar 5 2021, 3:52 PM
vlorentz added a commit: rDSTO2d540b0580cc: Add raw_extrinsic_metadata.id column in postgresql..Mar 22 2021, 1:55 PM
olasd mentioned this in T2520: Setup dedicated kafka cluster on new rocquencourt hardware.Mar 23 2021, 2:24 PM
olasd closed this task as Resolved.Mar 23 2021, 2:31 PM

After a lot of back and forth, and the release of swh.model v2.3.0 and swh.storage v0.26.0, this is now all done and deployed in staging and production.

We went the following way for the migration:

  • make swh.model write id fields in the journal
  • deploy swh.storage with the new swh.model (so all writes happen with the new model)
  • run swh storage backfill on the raw_extrinsic_metadata topic to fill the journal with objects using the new model
  • (make sure the journal gets compacted to remove old versions of the object, with a combination of topic.retention.ms and having to run the backfill multiple times for it to work on all the real-world data)
  • run swh storage replay on raw_extrinsic_metadata, using a fork of swh.storage that wrote objects to a new table (using the new schema)
  • once the replayer caught up, run some queries to spot check that all the data got properly migrated
  • once validated, stop the workers; stop replayer; deploy new version of swh.storage with new schema, move the new table in place of the old one (and take care of logical replication); then restart the workers
About · Developer information · Legal