Page MenuHomeSoftware Heritage

deposit.migrations: Migrate to latest storage api change
ClosedPublic

Authored by ardumont on Wed, Jul 29, 7:35 PM.

Details

Summary

It's code that no longer runs [1]. Just maintaining it along for the sake of
correctness.

[1] As all deposit migration script, it's a one time use production wise.

Related to T645

Test Plan

tox
(failing, needs storage release > 0.11.6)

Diff Detail

Repository
rDDEP Push deposit
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Wed, Jul 29, 7:35 PM

Build has FAILED

Patch application report for D3645 (id=12836)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit 693e8605e4edbbed44546386150fe87769115297
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/47/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/47/console

This feels like a function that should be in swh.storage.algos

ardumont edited the test plan for this revision. (Show Details)Wed, Jul 29, 8:18 PM

yeah...

Opening it to snapshot.algo, that will give me incentive to add tests on it ;)
Here i'm not too big on adding those on no longer used code.

ardumont planned changes to this revision.Thu, Jul 30, 9:58 AM
ardumont updated this revision to Diff 12841.Thu, Jul 30, 10:24 AM

Adapt according to swh.storage.snapshot.algos new algo function

ardumont updated this revision to Diff 12842.Thu, Jul 30, 10:25 AM

Adapt according to swh.storage.snapshot.algos new algo function

Build has FAILED

Patch application report for D3645 (id=12842)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit 0661eb608470b59726d276f6bf2a8c420f69b5ee
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/48/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/48/console

Build is green

Patch application report for D3645 (id=12841)

Could not rebase; Attempt merge onto 7667f7eef9...

Updating 7667f7ee..e026d33d
Fast-forward
 swh/storage/algos/snapshot.py            |  56 ++++++++-
 swh/storage/cassandra/cql.py             |  74 +++++++++++
 swh/storage/cassandra/storage.py         |  34 +++++
 swh/storage/db.py                        |  53 ++++----
 swh/storage/in_memory.py                 |  40 ++++++
 swh/storage/interface.py                 |  28 +++++
 swh/storage/storage.py                   |  47 +++++++
 swh/storage/tests/algos/test_snapshot.py |  64 +++++++++-
 swh/storage/tests/test_storage.py        | 205 +++++++++++++++++++++++++++++++
 9 files changed, 570 insertions(+), 31 deletions(-)
Changes applied before test
commit e026d33d08a488bc73f11f67ac1b8343cec5cc86
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jul 30 09:58:40 2020 +0200

    algos.snapshot: Open snapshot_get_from_revision
    
    This leverages the latest change in origin_visit_get and
    origin_visit_status_get to iterate over visit and visit status to detect a
    snapshot targetting a revision.
    
    This algo got used recently in the deposit. It may serve again.
    
    Related to T645

commit de11ef60ea74474c16df45149a97e9cfd8244ff0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 16:33:31 2020 +0200

    storage*: add origin_visit_status_get(...) -> PagedResult[OriginVisitStatus]
    
    Related to T645

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/615/ for more details.

ardumont updated this revision to Diff 12844.Thu, Jul 30, 11:19 AM

Rename according to change

Build has FAILED

Patch application report for D3645 (id=12844)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit e189378e1353d629d3d79a4e480379863622d68c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/49/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/49/console

douardda accepted this revision.Thu, Jul 30, 3:31 PM
This revision is now accepted and ready to land.Thu, Jul 30, 3:31 PM

Build has FAILED

Patch application report for D3645 (id=12844)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit e189378e1353d629d3d79a4e480379863622d68c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/50/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/50/console

Build is green

Patch application report for D3645 (id=12844)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit e189378e1353d629d3d79a4e480379863622d68c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/51/ for more details.