Page MenuHomeSoftware Heritage

Add support for raw_extrinsic_metadata in the replayer
ClosedPublic

Authored by olasd on Mar 10 2021, 9:37 AM.

Details

Summary

This also checks the basic raw_extrinsic_metadata codepaths in the
backfiller tests.

Depends on D5226

Test Plan

This whole pipeline has also been tested on some of the production data...

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D5227 (id=18715)

Could not rebase; Attempt merge onto 88ff2c2fa0...

Updating 88ff2c2f..7744a288
Fast-forward
 swh/storage/backfill.py                         | 77 +++++++++++++++++++++++++
 swh/storage/postgresql/converters.py            | 12 +++-
 swh/storage/replay.py                           | 17 +++++-
 swh/storage/tests/test_backfill.py              | 48 ++++++++++++++-
 swh/storage/tests/test_postgresql_converters.py | 36 +++++++++++-
 swh/storage/tests/test_replay.py                |  2 +
 6 files changed, 187 insertions(+), 5 deletions(-)
Changes applied before test
commit 7744a28823b9919f8180cc0dda5b6824e53e447c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:45:06 2021 +0100

    Add support for raw_extrinsic_metadata in the replayer
    
    This also checks the basic raw_extrinsic_metadata codepaths in the
    backfiller tests.

commit e8c785230f9a925ba3ee2cea4405797a9254d252
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:42:18 2021 +0100

    Add basic support for raw_extrinsic_metadata in the backfiller

commit 89ae0a148351c2dcb36d55f1e9cfaffa3158ec14
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:40:27 2021 +0100

    Add simple unit test for the backfill.byte_ranges function

commit 0d785d239eb9a53705c9e23fa183344a2606a5a5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:43:51 2021 +0100

    Add support for reading RawExtrinsicMetadata with raw URL targets
    
    We convert the target attribute to a hashed ExtendedSWHID before
    returning the object.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1200/ for more details.

This revision is now accepted and ready to land.Mar 10 2021, 1:30 PM

Build is green

Patch application report for D5227 (id=18728)

Could not rebase; Attempt merge onto 89ae0a1483...

Updating 89ae0a14..c4fdd6db
Fast-forward
 swh/storage/backfill.py            | 77 ++++++++++++++++++++++++++++++++++++++
 swh/storage/replay.py              | 17 ++++++++-
 swh/storage/tests/test_backfill.py | 25 +++++++++++++
 swh/storage/tests/test_replay.py   |  2 +
 4 files changed, 120 insertions(+), 1 deletion(-)
Changes applied before test
commit c4fdd6dbd0b3c04eefcdaa53a88d3aec3d0133f9
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:45:06 2021 +0100

    Add support for raw_extrinsic_metadata in the replayer
    
    This also checks the basic raw_extrinsic_metadata codepaths in the
    backfiller tests.

commit 53a58fa0335f7ca4a183c0073b0f52b19aa422aa
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Mar 9 17:42:18 2021 +0100

    Add basic support for raw_extrinsic_metadata in the backfiller

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1205/ for more details.