Page MenuHomeSoftware Heritage
Differential D3315

Migrate to swh.storage.algos.snapshot_get_latest
ClosedPublic
Actions

Authored by ardumont on Jun 18 2020, 5:50 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Commits
rDWAPPSd3ad49fe39d7: Migrate to swh.storage.algos.snapshot_get_latest
Summary

Depends on D3314

Test Plan

tox

Diff Detail

Repository
rDWAPPS Web applications
Branch
snapshot-get-latest
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 12957
Build 19741: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 19740: arc lint + arc unit

Event Timeline

ardumont created this revision.Jun 18 2020, 5:50 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 18 2020, 5:50 PM
ardumont edited the summary of this revision. (Show Details)Jun 18 2020, 5:50 PM
swh-public-ci added a comment.Jun 18 2020, 5:51 PM

Build has FAILED

Patch application report for D3315 (id=11735)

Rebasing onto c435bc30e4...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit f75fd0356bc4ef9a6530892bca610972a98e8e11
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/191/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/191/console

Harbormaster failed remote builds in B12945: Diff 11735!Jun 18 2020, 5:51 PM
ardumont mentioned this in D3314: Move deprecated endpoint snapshot_get_latest from api endpoint to algos.Jun 18 2020, 6:00 PM
ardumont added a child revision: D3316: service: Use latest origin visit status from an origin.Jun 18 2020, 6:43 PM
ardumont updated this revision to Diff 11738.Jun 19 2020, 9:40 AM

Account for the webapp use case which only wants a partial snapshot (for performance reason)

ardumont updated this revision to Diff 11740.Jun 19 2020, 9:46 AM

Update to require the partial snapshot as declared in the docstring

swh-public-ci added a comment.Jun 19 2020, 9:47 AM

Build has FAILED

Patch application report for D3315 (id=11740)

Rebasing onto c435bc30e4...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit ce1e5d5afd48c12d8e816176c6823c53d98b1414
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/194/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/194/console

Harbormaster failed remote builds in B12950: Diff 11740!Jun 19 2020, 9:47 AM
swh-public-ci added a comment.Jun 19 2020, 9:48 AM

Build is green

Patch application report for D3315 (id=11738)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..6bd727a
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  59 +++++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py | 110 +++++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 450 insertions(+), 295 deletions(-)
Changes applied before test
commit 6bd727a06677c0c60ffafcec1c3c5733d1f9c583
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/311/ for more details.

Harbormaster completed remote builds in B12948: Diff 11738.Jun 19 2020, 9:48 AM
ardumont updated this revision to Diff 11742.Jun 19 2020, 10:01 AM

Rebase on latest master

swh-public-ci added a comment.Jun 19 2020, 10:02 AM

Build has FAILED

Patch application report for D3315 (id=11742)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 32a9ca05e86bb2e43bdc58c426de09b86d0562b1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/195/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/195/console

Harbormaster failed remote builds in B12952: Diff 11742!Jun 19 2020, 10:02 AM
anlambert requested changes to this revision.Jun 19 2020, 11:27 AM
anlambert added a subscriber: anlambert.

LGTM, I added a couple of inline comments to address before I can accept the diff

swh/web/common/service.py
1027–1029

I would rather write

return converters.from_snapshot(snapshot.to_dict()) if snapshot is not None else None

or

if snapshot is not None:
     return converters.from_snapshot(snapshot.to_dict())
swh/web/tests/data.py
227–228

Use the snapshot model instead of a dict here (as you did in the remaining of that diff)

This revision now requires changes to proceed.Jun 19 2020, 11:27 AM
ardumont added inline comments.Jun 19 2020, 11:44 AM
swh/web/common/service.py
1027–1029

yeah, i'd prefer the first one, but it's too long about 1 char ;)

I'll rename to snp so it's fine ;)

the second i stopped using since when using mypy when there are types complain
about the missing return statement ¯\_(ツ)_/¯

I missed adding types to that function, I'll fix that if you don't mind ;)

swh/web/tests/data.py
227–228

I recall not doing it everywhere because i fought a lot initially ;)

The diff got simplified a lot once i found the right place to change.
I'll fix it.

ardumont updated this revision to Diff 11747.Jun 19 2020, 11:56 AM

Adapt according to review

swh-public-ci added a comment.Jun 19 2020, 11:57 AM

Build has FAILED

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 563e9658eb6c9f2bbbf96e53eb8607cae3d2808c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/197/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/197/console

Harbormaster failed remote builds in B12957: Diff 11747!Jun 19 2020, 11:57 AM
swh-public-ci added a comment.Jun 19 2020, 12:05 PM

Build has FAILED

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 48e21e65963217581b4521712e6d8ad79adbfe66
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/198/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/198/console

swh-public-ci added a comment.Jun 19 2020, 12:59 PM

Build is green

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 4fab1f6a69c6ebbc2aa511c5ba63ec20df026ce0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/200/ for more details.

Harbormaster completed remote builds in B12957: Diff 11747.Jun 19 2020, 12:59 PM
ardumont edited the test plan for this revision. (Show Details)Jun 19 2020, 1:00 PM
ardumont edited the summary of this revision. (Show Details)
ardumont added a child revision: D3318: misc/coverage: Add IPOL and NixOS logos.
anlambert accepted this revision.Jun 19 2020, 1:09 PM
This revision is now accepted and ready to land.Jun 19 2020, 1:09 PM
ardumont updated this revision to Diff 11749.Jun 19 2020, 1:13 PM
ardumont edited the summary of this revision. (Show Details)

Rebase on latest master

swh-public-ci added a comment.Jun 19 2020, 1:23 PM

Build is green

Patch application report for D3315 (id=11749)

Rebasing onto 0c7cbf061d...

Current branch diff-target is up to date.
Changes applied before test
commit d3ad49fe39d7d11ba1d22951da4936e7440a2e5a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/202/ for more details.

Harbormaster completed remote builds in B12959: Diff 11749.Jun 19 2020, 1:23 PM
Closed by commit rDWAPPSd3ad49fe39d7: Migrate to swh.storage.algos.snapshot_get_latest (authored by ardumont). · Explain WhyJun 19 2020, 1:24 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDWAPPSd3ad49fe39d7: Migrate to swh.storage.algos.snapshot_get_latest.
ardumont removed a child revision: D3318: misc/coverage: Add IPOL and NixOS logos.Jun 19 2020, 1:25 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
2 lines--
swh/
web/
common/
13 lines91%
Loading...
tests/
7 lines90%
Loading...
18 lines100%
Loading...
17 lines96%
Loading...

Diff 11747

View Options

requirements-swh.txt

Loading...
View Options

swh/web/common/service.py

Loading...
View Options

swh/web/tests/conftest.py

Loading...
View Options

swh/web/tests/data.py

Loading...
View Options

swh/web/tests/strategies.py

Loading...
About · Developer information · Legal