Page MenuHomeSoftware Heritage

Migrate to swh.storage.algos.snapshot_get_latest
ClosedPublic

Authored by ardumont on Jun 18 2020, 5:50 PM.

Details

Summary

Depends on D3314

Test Plan

tox

Diff Detail

Repository
rDWAPPS Web applications
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 12945
Build 19720: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 19719: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D3315 (id=11735)

Rebasing onto c435bc30e4...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit f75fd0356bc4ef9a6530892bca610972a98e8e11
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/191/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/191/console

Account for the webapp use case which only wants a partial snapshot (for performance reason)

Update to require the partial snapshot as declared in the docstring

Build has FAILED

Patch application report for D3315 (id=11740)

Rebasing onto c435bc30e4...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit ce1e5d5afd48c12d8e816176c6823c53d98b1414
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/194/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/194/console

Build is green

Patch application report for D3315 (id=11738)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..6bd727a
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  59 +++++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py | 110 +++++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 450 insertions(+), 295 deletions(-)
Changes applied before test
commit 6bd727a06677c0c60ffafcec1c3c5733d1f9c583
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/311/ for more details.

Build has FAILED

Patch application report for D3315 (id=11742)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 32a9ca05e86bb2e43bdc58c426de09b86d0562b1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/195/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/195/console

anlambert added a subscriber: anlambert.

LGTM, I added a couple of inline comments to address before I can accept the diff

swh/web/common/service.py
1025–1027

I would rather write

return converters.from_snapshot(snapshot.to_dict()) if snapshot is not None else None

or

if snapshot is not None:
     return converters.from_snapshot(snapshot.to_dict())
swh/web/tests/data.py
228

Use the snapshot model instead of a dict here (as you did in the remaining of that diff)

This revision now requires changes to proceed.Jun 19 2020, 11:27 AM
swh/web/common/service.py
1025–1027

yeah, i'd prefer the first one, but it's too long about 1 char ;)

I'll rename to snp so it's fine ;)

the second i stopped using since when using mypy when there are types complain
about the missing return statement ¯\_(ツ)_/¯

I missed adding types to that function, I'll fix that if you don't mind ;)

swh/web/tests/data.py
228

I recall not doing it everywhere because i fought a lot initially ;)

The diff got simplified a lot once i found the right place to change.
I'll fix it.

Build has FAILED

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 563e9658eb6c9f2bbbf96e53eb8607cae3d2808c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/197/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/197/console

Build has FAILED

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 48e21e65963217581b4521712e6d8ad79adbfe66
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/198/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/198/console

Build is green

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 4fab1f6a69c6ebbc2aa511c5ba63ec20df026ce0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/200/ for more details.

ardumont edited the summary of this revision. (Show Details)
This revision is now accepted and ready to land.Jun 19 2020, 1:09 PM
ardumont edited the summary of this revision. (Show Details)

Rebase on latest master

Build is green

Patch application report for D3315 (id=11749)

Rebasing onto 0c7cbf061d...

Current branch diff-target is up to date.
Changes applied before test
commit d3ad49fe39d7d11ba1d22951da4936e7440a2e5a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/202/ for more details.