Add an "index" for raw_extrinsic_metadata.id in swh.storage.cassandra
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	vlorentz
	Feb 2 2021, 1:40 PM

Description

As we want to query both by target and deduplicate by hash (+possibly query by hash too), we need two tables: the current one (access by target) and one to get a ref to a row of that table from a hash.

Revisions and Commits

rDSTO Storage manager
	D5030	rDSTOeff238371739 raw_extrinsic_metadata: Make (target, authority_id, discovery_date, fetcher_id)…

Related Objects
Search...

Status	Assigned	Task
Migrated	gitlab-migration	T2685 Test update of deposit with metadata-only in integration tests
Migrated	gitlab-migration	T3197 Mirror: fix common issues of a replayer session
Migrated	gitlab-migration	T3201 Mirror: unsupported Unicode escape sequence
Migrated	gitlab-migration	T2344 Build a connector for software deposit via Zenodo/InvenioRDM
Migrated	gitlab-migration	T2538 Add new option for the CLI swh-deposit for the metadata-only deposit
Migrated	gitlab-migration	T1021 SWORD deposit of metadata about an existing SWH object
Migrated	gitlab-migration	T2537 Extend new deposit endpoint to support metadata-only deposits
Migrated	gitlab-migration	T1258 Synthesize release objects for all upstream things that match the concept of a release
Migrated	gitlab-migration	T2059 Generate (swh) releases from all git tags
Migrated	gitlab-migration	T3089 Remove the 'metadata' column of the 'revision' table
Migrated	gitlab-migration	T2513 Copy metadata on revisions to the extrinsic metadata storage
Migrated	gitlab-migration	T3128 Improve deposit integration, management and display
Migrated	gitlab-migration	T2540 support the loading of metadata-only deposits in the metadata storage
Migrated	gitlab-migration	T2779 Put information (client, collection and deposit-id) inside metadata for metadata-only deposit
Migrated	gitlab-migration	T2520 Setup dedicated kafka cluster on new rocquencourt hardware
Migrated	gitlab-migration	T2686 Use hashes for all kafka keys
Migrated	gitlab-migration	T2668 Package loaders should write extrinsic metadata on directories instead of revisions/releases
Migrated	gitlab-migration	T2703 Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects
Migrated	gitlab-migration	T3018 Allow querying raw_extrinsic_metadata by hash in swh-storage
Migrated	gitlab-migration	T3022 Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields
Migrated	gitlab-migration	T3020 Add an "index" for raw_extrinsic_metadata.id in swh.storage.cassandra
Migrated	gitlab-migration	T3074 Migrate all packages away from the old SWHID class
Migrated	gitlab-migration	T3034 generalize usage of SWHID for referencing SWH archive objects

Event Timeline

vlorentz triaged this task as High priority.Feb 2 2021, 1:40 PM

vlorentz created this task.

vlorentz removed subscribers: rdicosmo, moranegg.

vlorentz renamed this task from Allow querying raw_extrinsic_metadata by hash in swh.storage.cassandra to Add an "index" for raw_extrinsic_metadata.id in swh.storage.cassandra.Feb 2 2021, 2:21 PM

vlorentz updated the task description. (Show Details)

vlorentz added a parent task: T3022: Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields.

vlorentz added a subtask: T3074: Migrate all packages away from the old SWHID class.Feb 26 2021, 6:57 PM

vlorentz closed subtask T3074: Migrate all packages away from the old SWHID class as Resolved.Mar 5 2021, 12:31 PM

vlorentz added a revision: D5030: raw_extrinsic_metadata: Make (target, authority_id, discovery_date, fetcher_id) non-unique.Mar 5 2021, 3:52 PM

vlorentz added a commit: rDSTOeff238371739: raw_extrinsic_metadata: Make (target, authority_id, discovery_date, fetcher_id)….Mar 22 2021, 1:55 PM

vlorentz closed this task as Resolved.Mar 23 2021, 2:32 PM