Page MenuHomeSoftware Heritage
Create Task
Maniphest T2513

Copy metadata on revisions to the extrinsic metadata storage
Closed, MigratedEdits Locked
Actions

Description

Previous runs of loaders didn't write to the metadata storage, only on revision objects

Revisions and Commits

rDSTO Storage manager
D6189rDSTO7dc2863ef1fa migrate_extrinsic_metadata: Add an option to limit the number of revisions
D3820rDSTO5ec70a6bdae1 Add a Python script to migrate extrinsic metadata from revision metadata.

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT3197 Mirror: fix common issues of a replayer session
 Migratedgitlab-migrationT3201 Mirror: unsupported Unicode escape sequence
 Migratedgitlab-migrationT1258 Synthesize release objects for all upstream things that match the concept of a release
 Migratedgitlab-migrationT2059 Generate (swh) releases from all git tags
 Migratedgitlab-migrationT3089 Remove the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT2513 Copy metadata on revisions to the extrinsic metadata storage
 Migratedgitlab-migrationT2512 Make all loaders write their extrinsic metadata to the appropriate storage
 Migratedgitlab-migrationT2496 Write deposit metadata for revisions in the generic metadata storage
 Migratedgitlab-migrationT2306 Generic storage for extrinsic, qualified metadata related to any node of the swh archive
 Migratedgitlab-migrationT2703 Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects
 Migratedgitlab-migrationT3017 Use hashes as keys in swh.journal.objects.raw_extrinsic_metadata
 Migratedgitlab-migrationT3018 Allow querying raw_extrinsic_metadata by hash in swh-storage
 Migratedgitlab-migrationT3022 Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields
 Migratedgitlab-migrationT3019 Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql
 Migratedgitlab-migrationT3020 Add an "index" for raw_extrinsic_metadata.id in swh.storage.cassandra
 Migratedgitlab-migrationT3074 Migrate all packages away from the old SWHID class
 Migratedgitlab-migrationT3034 generalize usage of SWHID for referencing SWH archive objects

Event Timeline

vlorentz triaged this task as Normal priority.Jul 29 2020, 7:31 PM
vlorentz created this task.
vlorentz added a subtask: T2512: Make all loaders write their extrinsic metadata to the appropriate storage.
vlorentz updated the task description. (Show Details)Jul 31 2020, 2:26 PM
vlorentz changed the status of subtask T2512: Make all loaders write their extrinsic metadata to the appropriate storage from Open to Work in Progress.Aug 17 2020, 2:38 PM
vlorentz added a revision: D3820: Add a Python script to migrate extrinsic metadata from revision metadata..Aug 20 2020, 3:25 PM
vlorentz changed the task status from Open to Work in Progress.Aug 28 2020, 1:20 PM
vlorentz moved this task from Backlog to Work in progress on the Roadmap 2020 board.
vlorentz added a commit: rDSTO5ec70a6bdae1: Add a Python script to migrate extrinsic metadata from revision metadata..Sep 10 2020, 10:07 AM
olasd mentioned this in T2582: Upgrade belvedere/main to PostgreSQL 12.Sep 10 2020, 10:17 AM
vlorentz closed subtask T2512: Make all loaders write their extrinsic metadata to the appropriate storage as Resolved.Sep 15 2020, 1:58 PM
moranegg moved this task from Backlog to Implementation on the Metadata workflow board.Sep 18 2020, 1:44 PM
moranegg added a subscriber: moranegg.Oct 1 2020, 11:40 AM

This task will take us one step towards a searchable archive :-)

We should keep a very conservative approach, I would suggest to keep the metadata and just copy.
this way, you don't need to distinguish between the fields that require and those that do not.

Finally, it will be less stressful to run a script that doesn't change the archive but is very useful for the search mechanisms we want to implement on the ERMDS (Extrinsic Raw MetaData Storage).

vlorentz added a comment.Oct 1 2020, 12:05 PM

Completely agree :)

vlorentz added a subtask: T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects.Nov 23 2020, 12:00 PM
vlorentz added a parent task: T3089: Remove the 'metadata' column of the 'revision' table.Mar 5 2021, 3:51 PM
vlorentz closed subtask T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects as Resolved.Mar 23 2021, 2:33 PM
vlorentz merged a task: T3185: Migrate extrinsic metadata from 'revision' to 'raw_extrinsic_metadata' tables.Apr 6 2021, 5:14 PM

https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/migrate_extrinsic_metadata.py

olasd claimed this task.Apr 6 2021, 7:07 PM
olasd added a subscriber: olasd.

The script is now running on getty.

olasd added a comment.Apr 6 2021, 7:57 PM

Tail of log:

Processed 0.46M rows (~0.2%, last revision: 0095624edf008b754fb1ed5bd656d22c63f984ff)
Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 1204, in <module>
    main(storage_dbconn, storage_url, deposit_dbconn, bytes.fromhex(first_id), True)
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 1165, in main
    handle_row(row, storage, deposit_cur, dry_run)
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 975, in handle_row
    storage, row["id"], metadata["original_artifact"][0]["filename"]
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 261, in pypi_origin_from_filename
    project_name = pypi_project_from_filename(filename)
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 252, in pypi_project_from_filename
    assert match, original_filename
AssertionError: pypops-201408-r4.tar.gz

I'll make the script log the revisions it's unable to process, rather than uselessly fall flat on its face.

olasd added a comment.Apr 6 2021, 8:12 PM

(I've also noticed dry_run was = True, so I fixed that as well :P)

olasd added a comment.Apr 6 2021, 10:20 PM
2021-04-06 20:19:19,898 __main__     ERROR    Could not parse revision metadata 00959a167bd98452c98ce73382f4b42179d53d32
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 1161, in main
    handle_row(row, storage, deposit_cur, dry_run)
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 979, in handle_row
    storage, row["id"], metadata["original_artifact"][0]["filename"]
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 265, in pypi_origin_from_filename
    project_name = pypi_project_from_filename(filename)
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 256, in pypi_project_from_filename
    assert match, original_filename
AssertionError: pypops-201408-r4.tar.gz
olasd added a comment.Apr 6 2021, 11:12 PM
2021-04-06 20:54:44,962 __main__     ERROR    Could not parse revision metadata 00c6e2fe046dee3b5ef629f74f4801345840e70a
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 1161, in main
    handle_row(row, storage, deposit_cur, dry_run)
  File "/usr/lib/python3/dist-packages/swh/storage/migrate_extrinsic_metadata.py", line 843, in handle_row
    assert "id" in actual_metadata or "title" in actual_metadata
AssertionError
vlorentz mentioned this in D5440: migrate_extrinsic_metadata: Allow 'atom:title' as alternative to 'title'.Apr 7 2021, 2:20 PM
vlorentz removed a parent task: T2202: Collect extrinsic metadata.Aug 19 2021, 9:20 AM
vlorentz added a revision: D6189: migrate_extrinsic_metadata: Add an option to limit the number of revisions.Sep 6 2021, 2:46 PM
vlorentz added a commit: rDSTO7dc2863ef1fa: migrate_extrinsic_metadata: Add an option to limit the number of revisions.Sep 9 2021, 3:27 PM
olasd added a comment.Oct 21 2022, 1:43 PM

I've relaunched the latest version of the migrate_extrinsic_metadata script on getty...

moranegg added a comment.Dec 6 2022, 10:48 PM

thanks @olasd for persevering, is there an ETA for the relaunch from October 21st?

gitlab-migration changed the status of subtask T2512: Make all loaders write their extrinsic metadata to the appropriate storage from Resolved to Migrated.Jan 8 2023, 10:00 PM
gitlab-migration changed the status of subtask T2703: Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects from Resolved to Migrated.
gitlab-migration closed this task as Migrated.Jan 8 2023, 10:22 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal