Page MenuHomeSoftware Heritage

Move deprecated endpoint snapshot_get_latest from api endpoint to algos
ClosedPublic

Authored by ardumont on Thu, Jun 18, 3:24 PM.

Details

Summary

This allows to avoid repeating the same pattern of retrieving the last snapshot
for a given origin.

Note that this also makes the new function return a Snapshot model object as
well.

Impacts only swh-web which I'll deal with [1]

(.../swh-environment) $ grep snapshot_get_latest */swh/**/*.py | grep -v "\btest" | grep -v "swh-storage/swh"
swh-web/swh/web/common/service.py:    snapshot = storage.snapshot_get_latest(origin, allowed_statuses)
`

[1] D3315

Related to T2310

Test Plan

tox

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Thu, Jun 18, 3:24 PM
ardumont edited the summary of this revision. (Show Details)Thu, Jun 18, 3:26 PM

Build has FAILED

Patch application report for D3314 (id=11732)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..912daad
Fast-forward
 swh/storage/algos/origin.py              |  51 ++++++-
 swh/storage/algos/snapshot.py            |  43 +++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py |  91 +++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 416 insertions(+), 295 deletions(-)
Changes applied before test
commit 912daad20f4896a778a2b3ee691b86c6dd5a7756
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/309/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/309/console

ardumont updated this revision to Diff 11733.Thu, Jun 18, 3:36 PM

Format (remove extra lines introduced for no reason, rename test methods...)

Build is green

Patch application report for D3314 (id=11733)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..077bd44
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  43 +++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py |  89 +++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 413 insertions(+), 295 deletions(-)
Changes applied before test
commit 077bd4408a96dcaa3e160ae312454c82d17e8e0c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/310/ for more details.

ardumont edited the summary of this revision. (Show Details)Thu, Jun 18, 6:00 PM
ardumont planned changes to this revision.Fri, Jun 19, 9:15 AM

Webapp requires a potentially limited snapshot (in terms of branches for good reason) so this must be possible to do it from this function as well.
Will adapt.

ardumont updated this revision to Diff 11739.Fri, Jun 19, 9:44 AM

Account for the webapp use case which deals with partial snapshot (for performance reason)

Build has FAILED

Patch application report for D3314 (id=11739)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..6bd727a
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  59 +++++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py | 110 +++++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 450 insertions(+), 295 deletions(-)
Changes applied before test
commit 6bd727a06677c0c60ffafcec1c3c5733d1f9c583
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/312/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/312/console

Build is green

Patch application report for D3314 (id=11739)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..6bd727a
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  59 +++++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py | 110 +++++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 450 insertions(+), 295 deletions(-)
Changes applied before test
commit 6bd727a06677c0c60ffafcec1c3c5733d1f9c583
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/313/ for more details.

olasd accepted this revision.Fri, Jun 19, 10:58 AM
olasd added a subscriber: olasd.

Thanks!

Just one comment inline.

swh/storage/algos/snapshot.py
79

Raise a ValueError instead?

This revision is now accepted and ready to land.Fri, Jun 19, 10:58 AM
ardumont added inline comments.Fri, Jun 19, 11:03 AM
swh/storage/algos/snapshot.py
79

Right!

ardumont updated this revision to Diff 11745.Fri, Jun 19, 11:19 AM

raise instead of assert

Build is green

Patch application report for D3314 (id=11745)

Rebasing onto 5480b7b6ca...

Current branch diff-target is up to date.
Changes applied before test
commit 16009078bcb90a6d598fe0076f15bb934d7abebb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/314/ for more details.