Page MenuHomeSoftware Heritage

Migrate to swh.storage.algos.snapshot_get_latest
ClosedPublic

Authored by ardumont on Jun 18 2020, 5:50 PM.

Details

Summary

Depends on D3314

Test Plan

tox

Diff Detail

Repository
rDWAPPS Web applications
Branch
snapshot-get-latest
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 12957
Build 19741: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 19740: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D3315 (id=11735)

Rebasing onto c435bc30e4...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit f75fd0356bc4ef9a6530892bca610972a98e8e11
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/191/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/191/console

Account for the webapp use case which only wants a partial snapshot (for performance reason)

Update to require the partial snapshot as declared in the docstring

Build has FAILED

Patch application report for D3315 (id=11740)

Rebasing onto c435bc30e4...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit ce1e5d5afd48c12d8e816176c6823c53d98b1414
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/194/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/194/console

Build is green

Patch application report for D3315 (id=11738)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..6bd727a
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  59 +++++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py | 110 +++++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 450 insertions(+), 295 deletions(-)
Changes applied before test
commit 6bd727a06677c0c60ffafcec1c3c5733d1f9c583
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/311/ for more details.

Build has FAILED

Patch application report for D3315 (id=11742)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 32a9ca05e86bb2e43bdc58c426de09b86d0562b1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/195/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/195/console

anlambert added a subscriber: anlambert.

LGTM, I added a couple of inline comments to address before I can accept the diff

swh/web/common/service.py
1027–1029

I would rather write

return converters.from_snapshot(snapshot.to_dict()) if snapshot is not None else None

or

if snapshot is not None:
     return converters.from_snapshot(snapshot.to_dict())
swh/web/tests/data.py
227–228

Use the snapshot model instead of a dict here (as you did in the remaining of that diff)

This revision now requires changes to proceed.Jun 19 2020, 11:27 AM
swh/web/common/service.py
1027–1029

yeah, i'd prefer the first one, but it's too long about 1 char ;)

I'll rename to snp so it's fine ;)

the second i stopped using since when using mypy when there are types complain
about the missing return statement ¯\_(ツ)_/¯

I missed adding types to that function, I'll fix that if you don't mind ;)

swh/web/tests/data.py
227–228

I recall not doing it everywhere because i fought a lot initially ;)

The diff got simplified a lot once i found the right place to change.
I'll fix it.

Build has FAILED

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 563e9658eb6c9f2bbbf96e53eb8607cae3d2808c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/197/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/197/console

Build has FAILED

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 48e21e65963217581b4521712e6d8ad79adbfe66
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/198/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/198/console

Build is green

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 4fab1f6a69c6ebbc2aa511c5ba63ec20df026ce0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/200/ for more details.

ardumont edited the summary of this revision. (Show Details)
This revision is now accepted and ready to land.Jun 19 2020, 1:09 PM
ardumont edited the summary of this revision. (Show Details)

Rebase on latest master

Build is green

Patch application report for D3315 (id=11749)

Rebasing onto 0c7cbf061d...

Current branch diff-target is up to date.
Changes applied before test
commit d3ad49fe39d7d11ba1d22951da4936e7440a2e5a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/202/ for more details.