Page MenuHomeSoftware Heritage
Differential D3681

storage*: Drop origin-get-range in favor of origin-list
ClosedPublic
Actions

Authored by ardumont on Aug 1 2020, 9:30 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T645: Type swh-storage endpoints with swh.model objects
Commits
rDSTOaa58e1092ffb: storage*: Drop origin-get-range in favor of origin-list
Summary

Remaining impact is swh-scheduler [1]:

$ grep -r 'iter_origins' $SWH_ENVIRONMENT_HOME/*/swh/* | grep -v "swh-storage"
Binary file /home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/swh/scheduler/cli/__pycache__/task.cpython-37.pyc matches
Binary file /home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/swh/scheduler/cli/__pycache__/task.cpython-37-pytest-6.0.1.pyc matches
/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/swh/scheduler/cli/task.py:    from swh.storage.algos.origin import iter_origins
/home/tony/work/inria/repo/swh/swh-environment/swh-scheduler/swh/scheduler/cli/task.py:    origins = iter_origins(storage, origin_from=min_id, origin_to=max_id)

[1] swh-web already dealt with in D3675

Related to T645

Depends on D3680

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Aug 1 2020, 9:30 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 1 2020, 9:30 AM
swh-public-ci added a comment.Aug 1 2020, 9:32 AM

Build has FAILED

Patch application report for D3681 (id=12961)

Could not rebase; Attempt merge onto 92f1183de0...

Updating 92f1183d..89e89239
Fast-forward
 swh/storage/algos/origin.py            | 30 ++++---------
 swh/storage/cassandra/storage.py       | 12 +++++-
 swh/storage/in_memory.py               | 24 ++++++-----
 swh/storage/interface.py               | 30 ++++++-------
 swh/storage/storage.py                 | 14 +++++-
 swh/storage/tests/algos/test_origin.py | 79 +++++-----------------------------
 swh/storage/tests/test_cassandra.py    |  4 --
 swh/storage/tests/test_storage.py      | 62 +++++++++-----------------
 8 files changed, 90 insertions(+), 165 deletions(-)
Changes applied before test
commit 89e892393ddbbf0fb34f82f93c30815f6c1a9fc5
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

commit 87c5ba223508b212ed6939496a6380a771b05e43
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 09:07:12 2020 +0200

    storage*: Do not allow unknown visit status in origin_visit*_get_latest
    
    That makes some storage (pg-storage) fail without that filtering.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/644/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/644/console

Harbormaster failed remote builds in B14199: Diff 12961!Aug 1 2020, 9:32 AM
ardumont edited the summary of this revision. (Show Details)Aug 1 2020, 9:33 AM
ardumont added a parent revision: D3680: storage*: Do not allow unknown visit status in origin_visit*_get_latest.
swh-public-ci added a comment.Aug 1 2020, 9:35 AM

Build has FAILED

Patch application report for D3681 (id=12961)

Could not rebase; Attempt merge onto 92f1183de0...

Updating 92f1183d..89e89239
Fast-forward
 swh/storage/algos/origin.py            | 30 ++++---------
 swh/storage/cassandra/storage.py       | 12 +++++-
 swh/storage/in_memory.py               | 24 ++++++-----
 swh/storage/interface.py               | 30 ++++++-------
 swh/storage/storage.py                 | 14 +++++-
 swh/storage/tests/algos/test_origin.py | 79 +++++-----------------------------
 swh/storage/tests/test_cassandra.py    |  4 --
 swh/storage/tests/test_storage.py      | 62 +++++++++-----------------
 8 files changed, 90 insertions(+), 165 deletions(-)
Changes applied before test
commit 89e892393ddbbf0fb34f82f93c30815f6c1a9fc5
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

commit 87c5ba223508b212ed6939496a6380a771b05e43
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 09:07:12 2020 +0200

    storage*: Do not allow unknown visit status in origin_visit*_get_latest
    
    That makes some storage (pg-storage) fail without that filtering.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/645/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/645/console

ardumont updated this revision to Diff 12962.Aug 1 2020, 9:35 AM

Fix flake8 issue about unused import (tests are fine otherwise)

swh-public-ci added a comment.Aug 1 2020, 9:43 AM

Build is green

Patch application report for D3681 (id=12962)

Could not rebase; Attempt merge onto 92f1183de0...

Updating 92f1183d..ad8ab5bd
Fast-forward
 swh/storage/algos/origin.py            | 30 ++++---------
 swh/storage/cassandra/storage.py       | 12 ++++-
 swh/storage/in_memory.py               | 24 +++++-----
 swh/storage/interface.py               | 30 +++++--------
 swh/storage/storage.py                 | 14 +++++-
 swh/storage/tests/algos/test_origin.py | 80 +++++-----------------------------
 swh/storage/tests/test_cassandra.py    |  4 --
 swh/storage/tests/test_storage.py      | 62 +++++++++-----------------
 8 files changed, 90 insertions(+), 166 deletions(-)
Changes applied before test
commit ad8ab5bd69ce85aa453cb920876f6e4f46fdd187
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

commit 87c5ba223508b212ed6939496a6380a771b05e43
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 09:07:12 2020 +0200

    storage*: Do not allow unknown visit status in origin_visit*_get_latest
    
    That makes some storage (pg-storage) fail without that filtering.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/646/ for more details.

Harbormaster completed remote builds in B14200: Diff 12962.Aug 1 2020, 9:43 AM
ardumont updated this revision to Diff 12963.Aug 1 2020, 9:48 AM

Simplify:

  • Drop missing skipped test instruction for cassandra
  • simplify test_origin assertions
  • Drop unneeded instruction in iter_origins function definition
swh-public-ci added a comment.Aug 1 2020, 9:56 AM

Build is green

Patch application report for D3681 (id=12963)

Could not rebase; Attempt merge onto 92f1183de0...

Updating 92f1183d..77757a0d
Fast-forward
 swh/storage/algos/origin.py            | 32 ++++---------
 swh/storage/cassandra/storage.py       | 12 ++++-
 swh/storage/in_memory.py               | 24 +++++-----
 swh/storage/interface.py               | 30 +++++-------
 swh/storage/storage.py                 | 14 +++++-
 swh/storage/tests/algos/test_origin.py | 84 +++++-----------------------------
 swh/storage/tests/test_cassandra.py    |  8 ----
 swh/storage/tests/test_storage.py      | 62 +++++++++----------------
 8 files changed, 90 insertions(+), 176 deletions(-)
Changes applied before test
commit 77757a0dccb429bd84154df11ad58c8442d75b79
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

commit 87c5ba223508b212ed6939496a6380a771b05e43
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 09:07:12 2020 +0200

    storage*: Do not allow unknown visit status in origin_visit*_get_latest
    
    That makes some storage (pg-storage) fail without that filtering.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/647/ for more details.

Harbormaster completed remote builds in B14201: Diff 12963.Aug 1 2020, 9:56 AM
ardumont mentioned this in D3682: cli.task: Migrate scheduler cli to latest storage change on iter_origins.Aug 1 2020, 10:39 AM
ardumont updated this revision to Diff 12971.Aug 1 2020, 6:25 PM

in-memory: Drop no longer needed self._origins_by_id state variable

swh-public-ci added a comment.Aug 1 2020, 6:30 PM

Build is green

Patch application report for D3681 (id=12971)

Could not rebase; Attempt merge onto 92f1183de0...

Updating 92f1183d..79b3f100
Fast-forward
 swh/storage/algos/origin.py            | 32 ++++---------
 swh/storage/cassandra/storage.py       | 12 ++++-
 swh/storage/in_memory.py               | 31 ++++++-------
 swh/storage/interface.py               | 30 +++++-------
 swh/storage/storage.py                 | 14 +++++-
 swh/storage/tests/algos/test_origin.py | 84 +++++-----------------------------
 swh/storage/tests/test_cassandra.py    |  8 ----
 swh/storage/tests/test_storage.py      | 62 +++++++++----------------
 8 files changed, 90 insertions(+), 183 deletions(-)
Changes applied before test
commit 79b3f1001186fbaa40ba3073e5e914be7e56eb9f
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

commit 87c5ba223508b212ed6939496a6380a771b05e43
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Aug 1 09:07:12 2020 +0200

    storage*: Do not allow unknown visit status in origin_visit*_get_latest
    
    That makes some storage (pg-storage) fail without that filtering.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/651/ for more details.

Harbormaster completed remote builds in B14209: Diff 12971.Aug 1 2020, 6:30 PM
vlorentz added a subscriber: vlorentz.Aug 3 2020, 9:10 AM
vlorentz added inline comments.
swh/storage/algos/origin.py
23–24

use stream_results

vlorentz requested changes to this revision.Aug 3 2020, 9:11 AM
This revision now requires changes to proceed.Aug 3 2020, 9:11 AM
ardumont added inline comments.Aug 3 2020, 10:38 AM
swh/storage/algos/origin.py
23–24

I cannot, can I?

D3682 needs it. That's why I opened this like this at least.

ardumont added inline comments.Aug 3 2020, 11:19 AM
swh/storage/algos/origin.py
23–24

Also, as mentioned in the scheduler diff, we can add here the logger.info about the page_token (or in stream_results directly). I don't really see which is best?

Right now, it's needed for the origins so i gather here is enough.

vlorentz added inline comments.Aug 3 2020, 11:21 AM
swh/storage/algos/origin.py
23–24

I think we should impelement iter_origins directly in the scheduler CLI as it has a rather specific need

ardumont added inline comments.Aug 3 2020, 11:33 AM
swh/storage/algos/origin.py
23–24

Ok, so here, just plain stream_results call as the origin-visit and status in the same module at the end.
And then inline this code in the scheduler diff instead

sounds like a plan to me.
If the needs becomes shareable, we'll adapt then.

Thanks for your input ;)

ardumont updated this revision to Diff 12984.Aug 3 2020, 11:39 AM

Adapt according to discussion

swh-public-ci added a comment.Aug 3 2020, 11:44 AM

Build has FAILED

Patch application report for D3681 (id=12984)

Rebasing onto 87c5ba2235...

Current branch diff-target is up to date.
Changes applied before test
commit aa58e1092ffb46f807cd232be5c25cefe01b6967
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/652/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/652/console

Harbormaster failed remote builds in B14225: Diff 12984!Aug 3 2020, 11:44 AM
vlorentz added inline comments.Aug 3 2020, 11:47 AM
swh/storage/algos/origin.py
23–24

👍

vlorentz accepted this revision.Aug 3 2020, 11:47 AM
This revision is now accepted and ready to land.Aug 3 2020, 11:47 AM
swh-public-ci added a comment.Aug 3 2020, 11:49 AM

Build is green

Patch application report for D3681 (id=12984)

Rebasing onto 87c5ba2235...

Current branch diff-target is up to date.
Changes applied before test
commit aa58e1092ffb46f807cd232be5c25cefe01b6967
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Jul 31 17:56:13 2020 +0200

    storage*: Drop origin-get-range in favor of origin-list
    
    Related to T645

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/653/ for more details.

Harbormaster completed remote builds in B14225: Diff 12984.Aug 3 2020, 11:49 AM
Closed by commit rDSTOaa58e1092ffb: storage*: Drop origin-get-range in favor of origin-list (authored by ardumont). · Explain WhyAug 3 2020, 11:53 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSTOaa58e1092ffb: storage*: Drop origin-get-range in favor of origin-list.

Revision Contents
Changeset List

PathSize
swh/
storage/
algos/
33 lines
17 lines
18 lines
tests/
algos/
44 lines
8 lines
36 lines

Diff 12988

View Options

swh/storage/algos/origin.py

Loading...
View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/interface.py

Loading...
View Options

swh/storage/tests/algos/test_origin.py

Loading...
View Options

swh/storage/tests/test_cassandra.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal