Page MenuHomeSoftware Heritage

Add basic support for raw_extrinsic_metadata in the backfiller
ClosedPublic

Authored by olasd on Mar 10 2021, 9:36 AM.

Details

Summary

Depends on D5225

Test Plan

needs the next diff to actually be tested

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D5226 (id=18714)

Could not rebase; Attempt merge onto 88ff2c2fa0...

Updating 88ff2c2f..e8c78523
Fast-forward
 swh/storage/backfill.py                         | 77 +++++++++++++++++++++++++
 swh/storage/postgresql/converters.py            | 12 +++-
 swh/storage/tests/test_backfill.py              | 38 +++++++++++-
 swh/storage/tests/test_postgresql_converters.py | 36 +++++++++++-
 4 files changed, 159 insertions(+), 4 deletions(-)
Changes applied before test
commit e8c785230f9a925ba3ee2cea4405797a9254d252
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:42:18 2021 +0100

    Add basic support for raw_extrinsic_metadata in the backfiller

commit 89ae0a148351c2dcb36d55f1e9cfaffa3158ec14
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:40:27 2021 +0100

    Add simple unit test for the backfill.byte_ranges function

commit 0d785d239eb9a53705c9e23fa183344a2606a5a5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:43:51 2021 +0100

    Add support for reading RawExtrinsicMetadata with raw URL targets
    
    We convert the target attribute to a hashed ExtendedSWHID before
    returning the object.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1199/ for more details.

vlorentz added inline comments.
swh/storage/backfill.py
401–402

I find this more readable, but no big deal

This revision is now accepted and ready to land.Mar 10 2021, 1:40 PM

Build was aborted

Patch application report for D5226 (id=18727)

Rebasing onto 89ae0a1483...

Current branch diff-target is up to date.
Changes applied before test
commit 53a58fa0335f7ca4a183c0073b0f52b19aa422aa
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:42:18 2021 +0100

    Add basic support for raw_extrinsic_metadata in the backfiller

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1204/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1204/console

Build is green

Patch application report for D5226 (id=18727)

Rebasing onto 89ae0a1483...

Current branch diff-target is up to date.
Changes applied before test
commit 53a58fa0335f7ca4a183c0073b0f52b19aa422aa
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:42:18 2021 +0100

    Add basic support for raw_extrinsic_metadata in the backfiller

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1206/ for more details.