Page MenuHomeSoftware Heritage
Differential D4905

Make PaginatedListedOriginList a concretization of PagedResult
ClosedPublic
Actions

Authored by vlorentz on Jan 21 2021, 1:08 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDSCHb93aa5be2c2d: Make PaginatedListedOriginList a concretization of PagedResult
Summary
  1. consistent with swh-storage and swh-indexer-storage
  2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

Diff Detail

Repository
rDSCH Scheduling utilities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jan 21 2021, 1:08 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJan 21 2021, 1:08 PM
swh-public-ci added a comment.Jan 21 2021, 1:08 PM

Build has FAILED

Patch application report for D4905 (id=17473)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..33c2606
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 4 files changed, 216 insertions(+), 90 deletions(-)
Changes applied before test
commit 33c2606cb1ea91a133460541aac271770e2a4fa9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/224/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/224/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 1:08 PM
Harbormaster failed remote builds in B18607: Diff 17473!
swh-public-ci added a comment.Jan 21 2021, 1:45 PM

Build has FAILED

Patch application report for D4905 (id=17473)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..33c2606
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 4 files changed, 216 insertions(+), 90 deletions(-)
Changes applied before test
commit 33c2606cb1ea91a133460541aac271770e2a4fa9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/228/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/228/console

vlorentz updated this revision to Diff 17475.Jan 21 2021, 2:27 PM

adapt missing code

swh-public-ci added a comment.Jan 21 2021, 2:29 PM

Build is green

Patch application report for D4905 (id=17475)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..b93aa5b
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 swh/scheduler/tests/test_simulator.py |   6 +-
 5 files changed, 219 insertions(+), 93 deletions(-)
Changes applied before test
commit b93aa5be2c2d5dc2130e1027698f3e1255052d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/229/ for more details.

Harbormaster completed remote builds in B18609: Diff 17475.Jan 21 2021, 2:29 PM
vlorentz requested review of this revision.Jan 21 2021, 2:29 PM
ardumont accepted this revision.Jan 21 2021, 4:50 PM
This revision is now accepted and ready to land.Jan 21 2021, 4:50 PM
Closed by commit rDSCHb93aa5be2c2d: Make PaginatedListedOriginList a concretization of PagedResult (authored by vlorentz). · Explain WhyJan 22 2021, 11:35 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSCHb93aa5be2c2d: Make PaginatedListedOriginList a concretization of PagedResult.

Revision Contents
Changeset List

PathSize
swh/
scheduler/
12 lines
30 lines
33 lines
tests/
19 lines
6 lines

Diff 17518

View Options

swh/scheduler/backend.py

Loading...
View Options

swh/scheduler/interface.py

Loading...
View Options

swh/scheduler/model.py

Loading...
View Options

swh/scheduler/tests/test_scheduler.py

Loading...
View Options

swh/scheduler/tests/test_simulator.py

Loading...
About · Developer information · Legal