Page MenuHomeSoftware Heritage
Create Task
Maniphest T2686

Use hashes for all kafka keys
Open, NormalPublic
Actions

Description

including origins, visits, visit statuses, and extrinsic metadata

currently we are using a mix of hashes, and dicts of bytes and str, so it's less than ideal.

It would also solve the issue we have to implement T2668.

Related Objects
Search...

Event Timeline

vlorentz triaged this task as Normal priority.Oct 12 2020, 1:04 PM
vlorentz created this task.
vlorentz added a parent task: T2520: Setup dedicated kafka cluster on new rocquencourt hardware.
vlorentz mentioned this in T2672: Inconsistent keys between visits and origin visits in the journal.
vlorentz updated the task description. (Show Details)
vlorentz added a parent task: T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.
vlorentz updated the task description. (Show Details)Oct 12 2020, 1:06 PM
vlorentz updated the task description. (Show Details)
vlorentz mentioned this in D4194: model: use visit ids in the unique key, instead of their date..Oct 12 2020, 1:56 PM
ardumont updated the task description. (Show Details)Oct 13 2020, 8:51 AM
vlorentz mentioned this in T2704: Use a hash as id/ unicity key for MetadataFetcher and MetadataAuthority.Oct 14 2020, 2:07 PM
vlorentz added subtasks: T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects, T2704: Use a hash as id/ unicity key for MetadataFetcher and MetadataAuthority.
vlorentz removed a parent task: T2668: Package loaders should write extrinsic metadata on directories instead of revisions/releases.
vlorentz closed subtask T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects as Resolved.Mar 23 2021, 2:33 PM
olasd added a comment.Mar 23 2021, 2:47 PM

The following objects remain:

  • origin
  • origin_visit
  • origin_visit_status
  • metadata_fetcher
  • metadata_authority

At this point, metadata_fetcher and metadata_authority are basically churn that should get entirely folded into raw_extrinsic_metadata (same as person in revision/release). origin_visit is mostly redundant with its origin_visit_statuses and could probably go away as well.

Then remains origin and origin_visit_status. Now that we have an ExtendedSWHID for origin, we can probably use that (one could argue that origin is redundant with origin_visit(_status)?, but we still do have some origins with zero visits). origin_visit_status could get a raw_extrinsic_metadata-like manifest.

vlorentz added a comment.Mar 23 2021, 2:55 PM

👍

(and we should keep the origin topic; we already have an ExtSWHID for origins anyway)

About · Developer information · Legal