Page MenuHomeSoftware Heritage
Differential D3315

Migrate to swh.storage.algos.snapshot_get_latest
ClosedPublic
Actions

Authored by ardumont on Jun 18 2020, 5:50 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Commits
rDWAPPSd3ad49fe39d7: Migrate to swh.storage.algos.snapshot_get_latest
Summary

Depends on D3314

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 12948
Build 19726: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 19725: arc lint + arc unit

Event Timeline

ardumont created this revision.Jun 18 2020, 5:50 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 18 2020, 5:50 PM
ardumont edited the summary of this revision. (Show Details)Jun 18 2020, 5:50 PM
swh-public-ci added a comment.Jun 18 2020, 5:51 PM

Build has FAILED

Patch application report for D3315 (id=11735)

Rebasing onto c435bc30e4...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit f75fd0356bc4ef9a6530892bca610972a98e8e11
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/191/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/191/console

Harbormaster failed remote builds in B12945: Diff 11735!Jun 18 2020, 5:51 PM
ardumont mentioned this in D3314: Move deprecated endpoint snapshot_get_latest from api endpoint to algos.Jun 18 2020, 6:00 PM
ardumont added a child revision: D3316: service: Use latest origin visit status from an origin.Jun 18 2020, 6:43 PM
ardumont updated this revision to Diff 11738.Jun 19 2020, 9:40 AM

Account for the webapp use case which only wants a partial snapshot (for performance reason)

ardumont updated this revision to Diff 11740.Jun 19 2020, 9:46 AM

Update to require the partial snapshot as declared in the docstring

swh-public-ci added a comment.Jun 19 2020, 9:47 AM

Build has FAILED

Patch application report for D3315 (id=11740)

Rebasing onto c435bc30e4...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit ce1e5d5afd48c12d8e816176c6823c53d98b1414
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/194/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/194/console

Harbormaster failed remote builds in B12950: Diff 11740!Jun 19 2020, 9:47 AM
swh-public-ci added a comment.Jun 19 2020, 9:48 AM

Build is green

Patch application report for D3315 (id=11738)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..6bd727a
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  59 +++++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py | 110 +++++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 450 insertions(+), 295 deletions(-)
Changes applied before test
commit 6bd727a06677c0c60ffafcec1c3c5733d1f9c583
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/311/ for more details.

Harbormaster completed remote builds in B12948: Diff 11738.Jun 19 2020, 9:48 AM
ardumont updated this revision to Diff 11742.Jun 19 2020, 10:01 AM

Rebase on latest master

swh-public-ci added a comment.Jun 19 2020, 10:02 AM

Build has FAILED

Patch application report for D3315 (id=11742)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 32a9ca05e86bb2e43bdc58c426de09b86d0562b1
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/195/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/195/console

Harbormaster failed remote builds in B12952: Diff 11742!Jun 19 2020, 10:02 AM
anlambert requested changes to this revision.Jun 19 2020, 11:27 AM
anlambert added a subscriber: anlambert.

LGTM, I added a couple of inline comments to address before I can accept the diff

swh/web/common/service.py
1026–1028 ↗(On Diff #11742)

I would rather write

return converters.from_snapshot(snapshot.to_dict()) if snapshot is not None else None

or

if snapshot is not None:
     return converters.from_snapshot(snapshot.to_dict())
swh/web/tests/data.py
228 ↗(On Diff #11742)

Use the snapshot model instead of a dict here (as you did in the remaining of that diff)

This revision now requires changes to proceed.Jun 19 2020, 11:27 AM
ardumont added inline comments.Jun 19 2020, 11:44 AM
swh/web/common/service.py
1026–1028 ↗(On Diff #11742)

yeah, i'd prefer the first one, but it's too long about 1 char ;)

I'll rename to snp so it's fine ;)

the second i stopped using since when using mypy when there are types complain
about the missing return statement ¯\_(ツ)_/¯

I missed adding types to that function, I'll fix that if you don't mind ;)

swh/web/tests/data.py
228 ↗(On Diff #11742)

I recall not doing it everywhere because i fought a lot initially ;)

The diff got simplified a lot once i found the right place to change.
I'll fix it.

ardumont updated this revision to Diff 11747.Jun 19 2020, 11:56 AM

Adapt according to review

swh-public-ci added a comment.Jun 19 2020, 11:57 AM

Build has FAILED

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 563e9658eb6c9f2bbbf96e53eb8607cae3d2808c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/197/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/197/console

Harbormaster failed remote builds in B12957: Diff 11747!Jun 19 2020, 11:57 AM
swh-public-ci added a comment.Jun 19 2020, 12:05 PM

Build has FAILED

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 48e21e65963217581b4521712e6d8ad79adbfe66
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/198/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/198/console

swh-public-ci added a comment.Jun 19 2020, 12:59 PM

Build is green

Patch application report for D3315 (id=11747)

Rebasing onto 0c7cbf061d...

First, rewinding head to replay your work on top of it...
Applying: Migrate to swh.storage.algos.snapshot_get_latest
Changes applied before test
commit 4fab1f6a69c6ebbc2aa511c5ba63ec20df026ce0
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/200/ for more details.

Harbormaster completed remote builds in B12957: Diff 11747.Jun 19 2020, 12:59 PM
ardumont edited the test plan for this revision. (Show Details)Jun 19 2020, 1:00 PM
ardumont edited the summary of this revision. (Show Details)
ardumont added a child revision: D3318: misc/coverage: Add IPOL and NixOS logos.
anlambert accepted this revision.Jun 19 2020, 1:09 PM
This revision is now accepted and ready to land.Jun 19 2020, 1:09 PM
ardumont updated this revision to Diff 11749.Jun 19 2020, 1:13 PM
ardumont edited the summary of this revision. (Show Details)

Rebase on latest master

swh-public-ci added a comment.Jun 19 2020, 1:23 PM

Build is green

Patch application report for D3315 (id=11749)

Rebasing onto 0c7cbf061d...

Current branch diff-target is up to date.
Changes applied before test
commit d3ad49fe39d7d11ba1d22951da4936e7440a2e5a
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 17:37:26 2020 +0200

    Migrate to swh.storage.algos.snapshot_get_latest
    
    Related to D3314

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/202/ for more details.

Harbormaster completed remote builds in B12959: Diff 11749.Jun 19 2020, 1:23 PM
Closed by commit rDWAPPSd3ad49fe39d7: Migrate to swh.storage.algos.snapshot_get_latest (authored by ardumont). · Explain WhyJun 19 2020, 1:24 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDWAPPSd3ad49fe39d7: Migrate to swh.storage.algos.snapshot_get_latest.
ardumont removed a child revision: D3318: misc/coverage: Add IPOL and NixOS logos.Jun 19 2020, 1:25 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
storage/
algos/
59 lines92%
Loading...
cassandra/
11 lines93%
Loading...
16 lines96%
Loading...
31 lines64%
Loading...
24 lines92%
Loading...
tests/
algos/
110 lines100%
Loading...
207 lines98%
Loading...

Diff 11738

View Options

swh/storage/algos/snapshot.py

Loading...
View Options

swh/storage/cassandra/storage.py

Loading...
View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/interface.py

Loading...
View Options

swh/storage/storage.py

Loading...
View Options

swh/storage/tests/algos/test_snapshot.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal