Details

Reviewers

olasd

Group Reviewers

Reviewers

Maniphest Tasks

T2310: Make origin visits immutable

Commits

rDSTO16009078bcb9: Move deprecated endpoint snapshot_get_latest from api endpoint to algos

Summary

This allows to avoid repeating the same pattern of retrieving the last snapshot
for a given origin.

Note that this also makes the new function return a Snapshot model object as
well.

Impacts only swh-web which I'll deal with [1]

(.../swh-environment) $ grep snapshot_get_latest */swh/**/*.py | grep -v "\btest" | grep -v "swh-storage/swh"
swh-web/swh/web/common/service.py:    snapshot = storage.snapshot_get_latest(origin, allowed_statuses)
`

[1] D3315

Related to T2310

Test Plan

tox

Diff Detail

Repository

rDSTO Storage manager

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 12943
Build 19716: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 19715: arc lint + arc unit

Event Timeline

ardumont created this revision.Jun 18 2020, 3:24 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJun 18 2020, 3:24 PM

ardumont edited the summary of this revision. (Show Details)Jun 18 2020, 3:26 PM

Build has FAILED

Patch application report for D3314 (id=11732)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..912daad
Fast-forward
 swh/storage/algos/origin.py              |  51 ++++++-
 swh/storage/algos/snapshot.py            |  43 +++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py |  91 +++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 416 insertions(+), 295 deletions(-)

Changes applied before test

commit 912daad20f4896a778a2b3ee691b86c6dd5a7756
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/309/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/309/console

Harbormaster failed remote builds in B12942: Diff 11732!Jun 18 2020, 3:32 PM

Format (remove extra lines introduced for no reason, rename test methods...)

Build is green

Patch application report for D3314 (id=11733)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..077bd44
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  43 +++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py |  89 +++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 413 insertions(+), 295 deletions(-)

Changes applied before test

commit 077bd4408a96dcaa3e160ae312454c82d17e8e0c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/310/ for more details.

Harbormaster completed remote builds in B12943: Diff 11733.Jun 18 2020, 3:45 PM

ardumont added a child revision: D3315: Migrate to swh.storage.algos.snapshot_get_latest.Jun 18 2020, 5:50 PM

ardumont edited the summary of this revision. (Show Details)Jun 18 2020, 6:00 PM

Webapp requires a potentially limited snapshot (in terms of branches for good reason) so this must be possible to do it from this function as well.
Will adapt.

Account for the webapp use case which deals with partial snapshot (for performance reason)

Build has FAILED

Patch application report for D3314 (id=11739)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..6bd727a
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  59 +++++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py | 110 +++++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 450 insertions(+), 295 deletions(-)

Changes applied before test

commit 6bd727a06677c0c60ffafcec1c3c5733d1f9c583
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/312/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/312/console

Harbormaster failed remote builds in B12949: Diff 11739!Jun 19 2020, 9:56 AM

Build is green

Patch application report for D3314 (id=11739)

Could not rebase; Attempt merge onto c49890105e...

Updating c498901..6bd727a
Fast-forward
 swh/storage/algos/origin.py              |  50 ++++++-
 swh/storage/algos/snapshot.py            |  59 +++++++-
 swh/storage/cassandra/storage.py         |  11 --
 swh/storage/in_memory.py                 |  16 ---
 swh/storage/interface.py                 |  31 ----
 swh/storage/storage.py                   |  24 ----
 swh/storage/tests/algos/test_origin.py   | 237 ++++++++++++++++++++++++++++++-
 swh/storage/tests/algos/test_snapshot.py | 110 +++++++++++++-
 swh/storage/tests/test_storage.py        | 207 ---------------------------
 9 files changed, 450 insertions(+), 295 deletions(-)

Changes applied before test

commit 6bd727a06677c0c60ffafcec1c3c5733d1f9c583
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

commit 5480b7b6ca18d8c38158ee54cfb567992461c357
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 13:40:16 2020 +0200

    algos.origin: Open origin-get-latest-visit-status function
    
    This will allow to avoid repeating the same pattern of retrieving the last
    visit status for a given origin.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/313/ for more details.

Harbormaster completed remote builds in B12949: Diff 11739.Jun 19 2020, 10:05 AM

Thanks!

Just one comment inline.

swh/storage/algos/snapshot.py
78	Raise a ValueError instead?

This revision is now accepted and ready to land.Jun 19 2020, 10:58 AM

ardumont added inline comments.Jun 19 2020, 11:03 AM

swh/storage/algos/snapshot.py
78	Right!

raise instead of assert

Build is green

Patch application report for D3314 (id=11745)

Rebasing onto 5480b7b6ca...

Current branch diff-target is up to date.

Changes applied before test

commit 16009078bcb90a6d598fe0076f15bb934d7abebb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Thu Jun 18 15:21:29 2020 +0200

    Move deprecated endpoint snapshot_get_latest from api endpoint to algos
    
    This allows to avoid repeating the same pattern of retrieving the last snapshot
    for a given origin.
    
    Note that this also makes the new function return a Snapshot model object as
    well.
    
    Related to T2310

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/314/ for more details.

Harbormaster completed remote builds in B12955: Diff 11745.Jun 19 2020, 11:27 AM

Closed by commit rDSTO16009078bcb9: Move deprecated endpoint snapshot_get_latest from api endpoint to algos (authored by ardumont). · Explain WhyJun 19 2020, 11:28 AM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSTO16009078bcb9: Move deprecated endpoint snapshot_get_latest from api endpoint to algos.

ardumont mentioned this in rDWAPPSd3ad49fe39d7: Migrate to swh.storage.algos.snapshot_get_latest.Jun 19 2020, 1:24 PM

ardumont mentioned this in T2310: Make origin visits immutable.Jun 23 2020, 2:37 PM

Diff 11733

View Options

swh/storage/algos/snapshot.py

Show First 20 Lines • Show All 137 Lines • ▼ Show 20 Lines	snapshots
object might track the commit where the master branch was pointing to at any		object might track the commit where the master branch was pointing to at any
given time, as well as the most recent release of a given package in the		given time, as well as the most recent release of a given package in the
stable suite of a FOSS distribution.		stable suite of a FOSS distribution.

visits		visits
links together software origins with snapshots. Every time an origin is		links together software origins with snapshots. Every time an origin is
consulted a new visit object is created, recording when (according to		consulted a new visit object is created, recording when (according to
Software Heritage clock) the visit happened and the full snapshot of the		Software Heritage clock) the visit happened and the full snapshot of the
state of the software origin at the time.		state of the software origin at the time.
ardumontAuthorUnsubmitted Done Inline Actions Those are currently mutable visits and indeed, it's not explicited. We create the first at the loading time. And we mutate its state when the loading is done to associate its visit status and snapshot. ardumont: Those are currently mutable visits and indeed, it's not explicited. We create the first at the…
ardumontAuthorUnsubmitted Done Inline Actions Historically, it got fixed recently [1], some visit updates were done in multiple steps. It's still in multiple steps but 2 only (1 creation, 1 update). [1] D2887 D2888 D2889 ardumont: Historically, it got fixed recently [1], some visit updates were done in multiple steps. It's…

		.. note::
		This model currently records visits as a single point in time. However, the
		actual visit process is not instantaneous. Loaders can record successive
		changes to the state of the visit, as their work progresses, as updates to
		the visit object.

Data structure		Data structure
--------------		--------------

.. _swh-merkle-dag:		.. _swh-merkle-dag:
.. figure:: images/swh-merkle-dag.svg		.. figure:: images/swh-merkle-dag.svg
:width: 1024px		:width: 1024px
:align: center		:align: center
▲ Show 20 Lines • Show All 102 Lines • Show Last 20 Lines

View Options

swh/storage/cassandra/storage.py

Show First 20 Lines • Show All 137 Lines • ▼ Show 20 Lines	snapshots
object might track the commit where the master branch was pointing to at any		object might track the commit where the master branch was pointing to at any
given time, as well as the most recent release of a given package in the		given time, as well as the most recent release of a given package in the
stable suite of a FOSS distribution.		stable suite of a FOSS distribution.

visits		visits
links together software origins with snapshots. Every time an origin is		links together software origins with snapshots. Every time an origin is
consulted a new visit object is created, recording when (according to		consulted a new visit object is created, recording when (according to
Software Heritage clock) the visit happened and the full snapshot of the		Software Heritage clock) the visit happened and the full snapshot of the
state of the software origin at the time.		state of the software origin at the time.
ardumontAuthorUnsubmitted Done Inline Actions Those are currently mutable visits and indeed, it's not explicited. We create the first at the loading time. And we mutate its state when the loading is done to associate its visit status and snapshot. ardumont: Those are currently mutable visits and indeed, it's not explicited. We create the first at the…
ardumontAuthorUnsubmitted Done Inline Actions Historically, it got fixed recently [1], some visit updates were done in multiple steps. It's still in multiple steps but 2 only (1 creation, 1 update). [1] D2887 D2888 D2889 ardumont: Historically, it got fixed recently [1], some visit updates were done in multiple steps. It's…

		.. note::
		This model currently records visits as a single point in time. However, the
		actual visit process is not instantaneous. Loaders can record successive
		changes to the state of the visit, as their work progresses, as updates to
		the visit object.

Data structure		Data structure
--------------		--------------

.. _swh-merkle-dag:		.. _swh-merkle-dag:
.. figure:: images/swh-merkle-dag.svg		.. figure:: images/swh-merkle-dag.svg
:width: 1024px		:width: 1024px
:align: center		:align: center
▲ Show 20 Lines • Show All 102 Lines • Show Last 20 Lines

View Options

swh/storage/in_memory.py

Show First 20 Lines • Show All 137 Lines • ▼ Show 20 Lines	snapshots
object might track the commit where the master branch was pointing to at any		object might track the commit where the master branch was pointing to at any
given time, as well as the most recent release of a given package in the		given time, as well as the most recent release of a given package in the
stable suite of a FOSS distribution.		stable suite of a FOSS distribution.

visits		visits
links together software origins with snapshots. Every time an origin is		links together software origins with snapshots. Every time an origin is
consulted a new visit object is created, recording when (according to		consulted a new visit object is created, recording when (according to
Software Heritage clock) the visit happened and the full snapshot of the		Software Heritage clock) the visit happened and the full snapshot of the
state of the software origin at the time.		state of the software origin at the time.
ardumontAuthorUnsubmitted Done Inline Actions Those are currently mutable visits and indeed, it's not explicited. We create the first at the loading time. And we mutate its state when the loading is done to associate its visit status and snapshot. ardumont: Those are currently mutable visits and indeed, it's not explicited. We create the first at the…
ardumontAuthorUnsubmitted Done Inline Actions Historically, it got fixed recently [1], some visit updates were done in multiple steps. It's still in multiple steps but 2 only (1 creation, 1 update). [1] D2887 D2888 D2889 ardumont: Historically, it got fixed recently [1], some visit updates were done in multiple steps. It's…

		.. note::
		This model currently records visits as a single point in time. However, the
		actual visit process is not instantaneous. Loaders can record successive
		changes to the state of the visit, as their work progresses, as updates to
		the visit object.

Data structure		Data structure
--------------		--------------

.. _swh-merkle-dag:		.. _swh-merkle-dag:
.. figure:: images/swh-merkle-dag.svg		.. figure:: images/swh-merkle-dag.svg
:width: 1024px		:width: 1024px
:align: center		:align: center
▲ Show 20 Lines • Show All 102 Lines • Show Last 20 Lines

View Options

swh/storage/interface.py

Show First 20 Lines • Show All 137 Lines • ▼ Show 20 Lines	snapshots
object might track the commit where the master branch was pointing to at any		object might track the commit where the master branch was pointing to at any
given time, as well as the most recent release of a given package in the		given time, as well as the most recent release of a given package in the
stable suite of a FOSS distribution.		stable suite of a FOSS distribution.

visits		visits
links together software origins with snapshots. Every time an origin is		links together software origins with snapshots. Every time an origin is
consulted a new visit object is created, recording when (according to		consulted a new visit object is created, recording when (according to
Software Heritage clock) the visit happened and the full snapshot of the		Software Heritage clock) the visit happened and the full snapshot of the
state of the software origin at the time.		state of the software origin at the time.
ardumontAuthorUnsubmitted Done Inline Actions Those are currently mutable visits and indeed, it's not explicited. We create the first at the loading time. And we mutate its state when the loading is done to associate its visit status and snapshot. ardumont: Those are currently mutable visits and indeed, it's not explicited. We create the first at the…
ardumontAuthorUnsubmitted Done Inline Actions Historically, it got fixed recently [1], some visit updates were done in multiple steps. It's still in multiple steps but 2 only (1 creation, 1 update). [1] D2887 D2888 D2889 ardumont: Historically, it got fixed recently [1], some visit updates were done in multiple steps. It's…

		.. note::
		This model currently records visits as a single point in time. However, the
		actual visit process is not instantaneous. Loaders can record successive
		changes to the state of the visit, as their work progresses, as updates to
		the visit object.

Data structure		Data structure
--------------		--------------

.. _swh-merkle-dag:		.. _swh-merkle-dag:
.. figure:: images/swh-merkle-dag.svg		.. figure:: images/swh-merkle-dag.svg
:width: 1024px		:width: 1024px
:align: center		:align: center
▲ Show 20 Lines • Show All 102 Lines • Show Last 20 Lines

View Options

swh/storage/storage.py

View Options

swh/storage/tests/algos/test_snapshot.py

View Options

swh/storage/tests/test_storage.py

Move deprecated endpoint snapshot_get_latest from api endpoint to algos
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Patch application report for D3314 (id=11732)

Changes applied before test

Patch application report for D3314 (id=11733)

Changes applied before test

Patch application report for D3314 (id=11739)

Changes applied before test

Patch application report for D3314 (id=11739)

Changes applied before test

Patch application report for D3314 (id=11745)

Changes applied before test

Revision Contents
Changeset List

Diff 11733

swh/storage/algos/snapshot.py

swh/storage/cassandra/storage.py

swh/storage/in_memory.py

swh/storage/interface.py

swh/storage/storage.py

swh/storage/tests/algos/test_snapshot.py

swh/storage/tests/test_storage.py

Move deprecated endpoint snapshot_get_latest from api endpoint to algosClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D3314 (id=11732)

Changes applied before test

Patch application report for D3314 (id=11733)

Changes applied before test

Patch application report for D3314 (id=11739)

Changes applied before test

Patch application report for D3314 (id=11739)

Changes applied before test

Patch application report for D3314 (id=11745)

Changes applied before test

Revision ContentsChangeset List

Diff 11733

swh/storage/algos/snapshot.py

swh/storage/cassandra/storage.py

swh/storage/in_memory.py

swh/storage/interface.py

swh/storage/storage.py

swh/storage/tests/algos/test_snapshot.py

swh/storage/tests/test_storage.py

Move deprecated endpoint snapshot_get_latest from api endpoint to algos
ClosedPublic
Actions

Revision Contents
Changeset List