Page MenuHomeSoftware Heritage
Differential D3645

deposit.migrations: Migrate to latest storage api change
ClosedPublic
Actions

Authored by ardumont on Jul 29 2020, 7:35 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T645: Type swh-storage endpoints with swh.model objects
Commits
rDDEPe189378e1353: deposit.migrations: Migrate to latest storage api change
Summary

It's code that no longer runs [1]. Just maintaining it along for the sake of
correctness.

[1] As all deposit migration script, it's a one time use production wise.

Related to T645

Test Plan

tox
(failing, needs storage release > 0.11.6)

Diff Detail

Repository
rDDEP Push deposit
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 14078
Build 21618: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 21617: arc lint + arc unit

Event Timeline

ardumont created this revision.Jul 29 2020, 7:35 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJul 29 2020, 7:35 PM
swh-public-ci added a comment.Jul 29 2020, 7:37 PM

Build has FAILED

Patch application report for D3645 (id=12836)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit 693e8605e4edbbed44546386150fe87769115297
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/47/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/47/console

Harbormaster failed remote builds in B14072: Diff 12836!Jul 29 2020, 7:37 PM
vlorentz added a subscriber: vlorentz.Jul 29 2020, 7:39 PM

This feels like a function that should be in swh.storage.algos

ardumont added a comment.Jul 29 2020, 7:54 PM

yeah...

ardumont edited the test plan for this revision. (Show Details)Jul 29 2020, 8:18 PM
ardumont mentioned this in D3627: storage*: origin_visit_get(...) -> PagedResult[OriginVisit].Jul 29 2020, 8:25 PM
ardumont added a comment.Jul 30 2020, 9:58 AM

yeah...

Opening it to snapshot.algo, that will give me incentive to add tests on it ;)
Here i'm not too big on adding those on no longer used code.

ardumont planned changes to this revision.Jul 30 2020, 9:58 AM
ardumont mentioned this in D3648: algos.snapshot: Open snapshot_get_from_revision algorithm.Jul 30 2020, 10:02 AM
ardumont updated this revision to Diff 12841.Jul 30 2020, 10:24 AM

Adapt according to swh.storage.snapshot.algos new algo function

ardumont updated this revision to Diff 12842.Jul 30 2020, 10:25 AM

Adapt according to swh.storage.snapshot.algos new algo function

swh-public-ci added a comment.Jul 30 2020, 10:26 AM

Build has FAILED

Patch application report for D3645 (id=12842)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit 0661eb608470b59726d276f6bf2a8c420f69b5ee
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/48/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/48/console

Harbormaster failed remote builds in B14078: Diff 12842!Jul 30 2020, 10:26 AM
swh-public-ci added a comment.Jul 30 2020, 10:34 AM

Build is green

Patch application report for D3645 (id=12841)

Could not rebase; Attempt merge onto 7667f7eef9...

Updating 7667f7ee..e026d33d
Fast-forward
 swh/storage/algos/snapshot.py            |  56 ++++++++-
 swh/storage/cassandra/cql.py             |  74 +++++++++++
 swh/storage/cassandra/storage.py         |  34 +++++
 swh/storage/db.py                        |  53 ++++----
 swh/storage/in_memory.py                 |  40 ++++++
 swh/storage/interface.py                 |  28 +++++
 swh/storage/storage.py                   |  47 +++++++
 swh/storage/tests/algos/test_snapshot.py |  64 +++++++++-
 swh/storage/tests/test_storage.py        | 205 +++++++++++++++++++++++++++++++
 9 files changed, 570 insertions(+), 31 deletions(-)
Changes applied before test
commit e026d33d08a488bc73f11f67ac1b8343cec5cc86
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jul 30 09:58:40 2020 +0200

    algos.snapshot: Open snapshot_get_from_revision
    
    This leverages the latest change in origin_visit_get and
    origin_visit_status_get to iterate over visit and visit status to detect a
    snapshot targetting a revision.
    
    This algo got used recently in the deposit. It may serve again.
    
    Related to T645

commit de11ef60ea74474c16df45149a97e9cfd8244ff0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 16:33:31 2020 +0200

    storage*: add origin_visit_status_get(...) -> PagedResult[OriginVisitStatus]
    
    Related to T645

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/615/ for more details.

Harbormaster completed remote builds in B14077: Diff 12841.Jul 30 2020, 10:34 AM
ardumont updated this revision to Diff 12844.Jul 30 2020, 11:19 AM

Rename according to change

swh-public-ci added a comment.Jul 30 2020, 11:21 AM

Build has FAILED

Patch application report for D3645 (id=12844)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit e189378e1353d629d3d79a4e480379863622d68c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/49/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/49/console

Harbormaster failed remote builds in B14080: Diff 12844!Jul 30 2020, 11:21 AM
douardda accepted this revision.Jul 30 2020, 3:31 PM
This revision is now accepted and ready to land.Jul 30 2020, 3:31 PM
swh-public-ci added a comment.Jul 30 2020, 4:32 PM

Build has FAILED

Patch application report for D3645 (id=12844)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit e189378e1353d629d3d79a4e480379863622d68c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/50/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/50/console

swh-public-ci added a comment.Jul 30 2020, 4:55 PM

Build is green

Patch application report for D3645 (id=12844)

Rebasing onto 402e4b83bb...

Current branch diff-target is up to date.
Changes applied before test
commit e189378e1353d629d3d79a4e480379863622d68c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Wed Jul 29 19:32:46 2020 +0200

    deposit.migrations: Migrate to latest storage api change
    
    It's code that no longer runs. Just maintaining it along for the sake of
    correctness.
    
    Related to T645

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/51/ for more details.

Harbormaster completed remote builds in B14080: Diff 12844.Jul 30 2020, 4:56 PM
Closed by commit rDDEPe189378e1353: deposit.migrations: Migrate to latest storage api change (authored by ardumont). · Explain WhyJul 30 2020, 4:56 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDDEPe189378e1353: deposit.migrations: Migrate to latest storage api change.

Revision Contents
Changeset List

PathSize
swh/
deposit/
migrations/
42 lines

Diff 12842

View Options

swh/deposit/migrations/0018_migrate_swhids.py

Loading...
About · Developer information · Legal