Page MenuHomeSoftware Heritage

Make PaginatedListedOriginList a concretization of PagedResult
ClosedPublic

Authored by vlorentz on Jan 21 2021, 1:08 PM.

Details

Summary
  1. consistent with swh-storage and swh-indexer-storage
  2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
scheduling-policy
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18609
Build 28783: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28782: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D4905 (id=17473)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..33c2606
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 4 files changed, 216 insertions(+), 90 deletions(-)
Changes applied before test
commit 33c2606cb1ea91a133460541aac271770e2a4fa9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/224/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/224/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 1:08 PM
Harbormaster failed remote builds in B18607: Diff 17473!

Build has FAILED

Patch application report for D4905 (id=17473)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..33c2606
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 4 files changed, 216 insertions(+), 90 deletions(-)
Changes applied before test
commit 33c2606cb1ea91a133460541aac271770e2a4fa9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/228/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/228/console

Build is green

Patch application report for D4905 (id=17475)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..b93aa5b
Fast-forward
 swh/scheduler/backend.py              |  46 ++++++--
 swh/scheduler/interface.py            |  30 ++++--
 swh/scheduler/model.py                |  33 +-----
 swh/scheduler/tests/test_scheduler.py | 197 +++++++++++++++++++++++++++-------
 swh/scheduler/tests/test_simulator.py |   6 +-
 5 files changed, 219 insertions(+), 93 deletions(-)
Changes applied before test
commit b93aa5be2c2d5dc2130e1027698f3e1255052d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/229/ for more details.

This revision is now accepted and ready to land.Jan 21 2021, 4:50 PM