Page MenuHomeSoftware Heritage
Create Task
Maniphest T2703

Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects
Closed, MigratedEdits Locked
Actions

Description

It turns out that using only swhid+authority_id+discovery_date+fetcher_id is not a good unicity key for RawExtrinsicMetadata objects, as we already need to include some other fields in it (context keys, see T2668); and we might need to include others in the future.

Additionally, it is currently intentionally unspecified in the spec and in the RPC interface what happens if we write two different objects with the same key.
This is fine, but less than ideal.

Hashing the entire object solves both these issues.

The only drawback is that the unicity key isn't human-readable anymore, and requires an API request to know what SWHID it's about. But we are already doing that for most objects, and I don't think it matters much anyway.

Revisions and Commits

rDMOD Data model
D4970rDMODfc808e1fc9e5 model: Add 'id' field to RawExtrinsicMetadata
D4935rDMODf6eab95253f1 identifiers: Add raw_extrinsic_metadata_identifier
D4348rDMOD9da17a51dfaf Rename the RawExtrinsicMetadata id field to target
D4307rDMOD498a1076416a Update the HashableObject interface to take the object itself

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT2685 Test update of deposit with metadata-only in integration tests
 Migratedgitlab-migrationT3197 Mirror: fix common issues of a replayer session
 Migratedgitlab-migrationT3201 Mirror: unsupported Unicode escape sequence
 Migratedgitlab-migrationT2344 Build a connector for software deposit via Zenodo/InvenioRDM
 Migratedgitlab-migrationT2538 Add new option for the CLI swh-deposit for the metadata-only deposit
 Migratedgitlab-migrationT1021 SWORD deposit of metadata about an existing SWH object
 Migratedgitlab-migrationT2537 Extend new deposit endpoint to support metadata-only deposits
 Migratedgitlab-migrationT1258 Synthesize release objects for all upstream things that match the concept of a release
 Migratedgitlab-migrationT2059 Generate (swh) releases from all git tags
 Migratedgitlab-migrationT3089 Remove the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT2513 Copy metadata on revisions to the extrinsic metadata storage
 Migratedgitlab-migrationT3128 Improve deposit integration, management and display
 Migratedgitlab-migrationT2540 support the loading of metadata-only deposits in the metadata storage
 Migratedgitlab-migrationT2779 Put information (client, collection and deposit-id) inside metadata for metadata-only deposit
 Migratedgitlab-migrationT2520 Setup dedicated kafka cluster on new rocquencourt hardware
 Migratedgitlab-migrationT2686 Use hashes for all kafka keys
 Migratedgitlab-migrationT2668 Package loaders should write extrinsic metadata on directories instead of revisions/releases
 Migratedgitlab-migrationT2703 Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects
 Migratedgitlab-migrationT3017 Use hashes as keys in swh.journal.objects.raw_extrinsic_metadata
 Migratedgitlab-migrationT3018 Allow querying raw_extrinsic_metadata by hash in swh-storage
 Migratedgitlab-migrationT3022 Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields
 Migratedgitlab-migrationT3019 Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql
 Migratedgitlab-migrationT3020 Add an "index" for raw_extrinsic_metadata.id in swh.storage.cassandra
 Migratedgitlab-migrationT3074 Migrate all packages away from the old SWHID class
 Migratedgitlab-migrationT3034 generalize usage of SWHID for referencing SWH archive objects

Event Timeline

vlorentz renamed this task from Use intrinsic identifiers for RawExtrinsicMetadata objects to Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects.Oct 14 2020, 2:01 PM
vlorentz triaged this task as High priority.
vlorentz created this task.
vlorentz edited projects, added Data Model; removed Package Loader.
vlorentz added a parent task: T2686: Use hashes for all kafka keys.Oct 14 2020, 2:08 PM
olasd added a revision: D4307: Update the HashableObject interface to take the object itself.Oct 19 2020, 4:26 PM
olasd added a commit: rDMOD498a1076416a: Update the HashableObject interface to take the object itself.Oct 23 2020, 5:15 PM
olasd added a revision: D4348: Rename the RawExtrinsicMetadata id field to target.Oct 23 2020, 5:20 PM
olasd added a commit: rDMOD9da17a51dfaf: Rename the RawExtrinsicMetadata id field to target.Oct 27 2020, 10:36 AM
olasd mentioned this in T2749: Clean up raw_extrinsic_metadata table.Nov 2 2020, 3:05 PM
vlorentz added a parent task: T2513: Copy metadata on revisions to the extrinsic metadata storage.Nov 23 2020, 12:00 PM
vlorentz mentioned this in T2779: Put information (client, collection and deposit-id) inside metadata for metadata-only deposit.Jan 4 2021, 2:24 PM
vlorentz added a comment.EditedJan 5 2021, 3:20 PM

Proposed manifest format:

type: $ValueOfMetadataTargetType
target: $UrlOrSwhid
discovery_date: $ISO8601
authority: $NameIncludingSpaces <$Url>
fetcher: $NameIncludingSpaces $Version
format: $Str
origin: $Str  <- optional
visit: $IntInDecimal  <- optional
snapshot: $Swhid  <- optional
release: $Swhid  <- optional
revision: $Swhid  <- optional
path: $BytesWithEscapedNewLines  <-- optional
directory: $Swhid  <- optional

$MetadataBytes

Note that the values of MetadataTargetType are SWH names (eg. "revision"), not git names (eg. "commit").

vlorentz added a parent task: T2779: Put information (client, collection and deposit-id) inside metadata for metadata-only deposit.Jan 7 2021, 1:53 PM
olasd added a comment.Jan 7 2021, 8:18 PM

Sounds good to me.

vlorentz claimed this task.Jan 11 2021, 4:27 PM
vlorentz mentioned this in D4935: identifiers: Add raw_extrinsic_metadata_identifier.Jan 25 2021, 12:31 PM
vlorentz added a revision: D4935: identifiers: Add raw_extrinsic_metadata_identifier.Jan 25 2021, 12:32 PM
ardumont mentioned this in D4970: model: Add 'id' field to RawExtrinsicMetadata.Feb 2 2021, 2:00 PM
vlorentz added a revision: D4970: model: Add 'id' field to RawExtrinsicMetadata.Feb 3 2021, 9:13 AM
vlorentz added a commit: rDMODf6eab95253f1: identifiers: Add raw_extrinsic_metadata_identifier.Mar 4 2021, 3:10 PM
vlorentz added a commit: rDMODfc808e1fc9e5: model: Add 'id' field to RawExtrinsicMetadata.
olasd closed subtask T3022: Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields as Resolved.Mar 23 2021, 2:25 PM
vlorentz closed this task as Resolved.Mar 23 2021, 2:33 PM
vlorentz closed subtask T3017: Use hashes as keys in swh.journal.objects.raw_extrinsic_metadata as Resolved.
vlorentz closed subtask T3018: Allow querying raw_extrinsic_metadata by hash in swh-storage as Resolved.Sep 3 2021, 11:38 AM
gitlab-migration changed the status of subtask T3018: Allow querying raw_extrinsic_metadata by hash in swh-storage from Resolved to Migrated.Jan 8 2023, 4:33 PM
gitlab-migration changed the status of subtask T3022: Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields from Resolved to Migrated.
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 10:01 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T3017: Use hashes as keys in swh.journal.objects.raw_extrinsic_metadata from Resolved to Migrated.Jan 8 2023, 10:01 PM
About · Developer information · Legal