For now, this process only writes a log every simulated day.
Diff Detail
Diff Detail
- Repository
- rDSCH Scheduling utilities
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 18626 Build 28811: Phabricator diff pipeline on jenkins Jenkins console · Jenkins Build 28810: arc lint + arc unit
Event Timeline
Comment Actions
Build is green
Patch application report for D4913 (id=17492)
Could not rebase; Attempt merge onto 03460207a1...
Updating 0346020..8e7377d Fast-forward swh/scheduler/backend.py | 107 ++++++++++++++++++++++------ swh/scheduler/interface.py | 45 +++++++++--- swh/scheduler/model.py | 33 +-------- swh/scheduler/simulator/__init__.py | 25 ++++--- swh/scheduler/simulator/origin_scheduler.py | 2 +- swh/scheduler/simulator/origins.py | 83 ++++++++++++++++++++- swh/scheduler/tests/test_scheduler.py | 93 ++++++++++++++++++++---- swh/scheduler/tests/test_simulator.py | 6 +- 8 files changed, 301 insertions(+), 93 deletions(-)
Changes applied before test
commit 8e7377d8af45ef8e8234b57dc6a16be75dd74ac5 Author: Valentin Lorentz <vlorentz@softwareheritage.org> Date: Thu Jan 21 17:38:41 2021 +0100 simulator: add a trivial heartbeat process to show progress For now, this process only writes a log every simulated day. commit ba303f946ecd3e15e58de0072ce71b50aa423d59 Author: Nicolas Dandrimont <nicolas@dandrimont.eu> Date: Thu Jan 21 17:31:43 2021 +0100 grab_next_visits: don't re-schedule visits too fast The earlier implementation would just schedule new visits for origins forever, regardless of whether they were already scheduled or not. commit 808ae6851faee9b633e773f9150d360cdb927146 Author: Nicolas Dandrimont <nicolas@dandrimont.eu> Date: Thu Jan 21 17:29:45 2021 +0100 Allow overriding the timestamp of grab_next_visits This makes the simulator behavior more consistent with reality. commit 9943195d31c51a44325cba09d07fb6e904d45a00 Author: Nicolas Dandrimont <nicolas@dandrimont.eu> Date: Thu Jan 21 17:27:40 2021 +0100 Construct grab_next_visits query arguments incrementally commit 72070b7bf628788b6872e90a3f8ac8f0c01b70d9 Author: Valentin Lorentz <vlorentz@softwareheritage.org> Date: Thu Jan 21 14:57:42 2021 +0100 simulator: add simple lister simulation commit 1f1aad459c4b0740ecbe96e9809e4b31f66bf999 Author: Valentin Lorentz <vlorentz@softwareheritage.org> Date: Thu Jan 21 14:54:53 2021 +0100 Factor out ListedOrigin generation to use the OriginModel This generates consistent last_update values according to the model and simulated time. commit b93aa5be2c2d5dc2130e1027698f3e1255052d8d Author: Valentin Lorentz <vlorentz@softwareheritage.org> Date: Thu Jan 21 13:01:53 2021 +0100 Make PaginatedListedOriginList a concretization of PagedResult 1. consistent with swh-storage and swh-indexer-storage 2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins. commit 2f47936731cf438a5195978a2af3250597b693b5 Author: Nicolas Dandrimont <nicolas@dandrimont.eu> Date: Wed Jan 20 17:29:16 2021 +0100 Add scheduling policy for already visited origins with known last update This policy schedules origins by decreasing order of "visit lag" (that is, origins with the most lag are scheduled first). commit acad712ad3f71f88f99e45e9b4f571ad751945dc Author: Nicolas Dandrimont <nicolas@dandrimont.eu> Date: Wed Jan 20 17:25:46 2021 +0100 Add scheduling policy for never visited origins This policy orders never visited origins by increasing date of last update (scheduling the "oldest" never visited origins first).
See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/235/ for more details.