Differential D4905

Make PaginatedListedOriginList a concretization of PagedResult
ClosedPublic
Actions

Authored by vlorentz on Jan 21 2021, 1:08 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDSCHb93aa5be2c2d: Make PaginatedListedOriginList a concretization of PagedResult

Summary

consistent with swh-storage and swh-indexer-storage
we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

Diff Detail

Repository

rDSCH Scheduling utilities

Branch

scheduling-policy

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 18609
Build 28783: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 28782: arc lint + arc unit

Event Timeline

vlorentz created this revision.Jan 21 2021, 1:08 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 21 2021, 1:08 PM

Build has FAILED

Patch application report for D4905 (id=17473)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..33c2606
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 4 files changed, 216 insertions(+), 90 deletions(-)

Changes applied before test

commit 33c2606cb1ea91a133460541aac271770e2a4fa9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/224/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/224/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 1:08 PM

Harbormaster failed remote builds in B18607: Diff 17473!

Build has FAILED

Patch application report for D4905 (id=17473)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..33c2606
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 4 files changed, 216 insertions(+), 90 deletions(-)

Changes applied before test

commit 33c2606cb1ea91a133460541aac271770e2a4fa9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/228/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/228/console

adapt missing code

Build is green

Patch application report for D4905 (id=17475)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..b93aa5b
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 swh/scheduler/tests/test_simulator.py |   6 +-
 5 files changed, 219 insertions(+), 93 deletions(-)

Changes applied before test

commit b93aa5be2c2d5dc2130e1027698f3e1255052d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/229/ for more details.

Harbormaster completed remote builds in B18609: Diff 17475.Jan 21 2021, 2:29 PM

vlorentz requested review of this revision.Jan 21 2021, 2:29 PM

ardumont accepted this revision.Jan 21 2021, 4:50 PM

This revision is now accepted and ready to land.Jan 21 2021, 4:50 PM

Closed by commit rDSCHb93aa5be2c2d: Make PaginatedListedOriginList a concretization of PagedResult (authored by vlorentz). · Explain WhyJan 22 2021, 11:35 AM

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDSCHb93aa5be2c2d: Make PaginatedListedOriginList a concretization of PagedResult.