Page MenuHomeSoftware Heritage
Differential D4911

Allow overriding the timestamp of grab_next_visits
ClosedPublic
Actions

Authored by vlorentz on Jan 21 2021, 6:07 PM.

Details

Reviewers
olasd
douardda
Group Reviewers
Reviewers
Commits
rDSCH2b39cbcabf99: Allow overriding the timestamp of grab_next_visits
Summary

This makes the simulator behavior more consistent with reality.

Diff Detail

Event Timeline

vlorentz created this revision.Jan 21 2021, 6:07 PM
swh-public-ci added a comment.Jan 21 2021, 6:13 PM

Build has FAILED

Patch application report for D4911 (id=17490)

Could not rebase; Attempt merge onto 03460207a1...

Updating 0346020..808ae68
Fast-forward
 swh/scheduler/backend.py                    | 90 ++++++++++++++++++++++------
 swh/scheduler/interface.py                  | 45 ++++++++++----
 swh/scheduler/model.py                      | 33 +---------
 swh/scheduler/simulator/__init__.py         | 18 +++---
 swh/scheduler/simulator/origin_scheduler.py |  2 +-
 swh/scheduler/simulator/origins.py          | 83 ++++++++++++++++++++++++-
 swh/scheduler/tests/test_scheduler.py       | 93 +++++++++++++++++++++++++----
 swh/scheduler/tests/test_simulator.py       |  6 +-
 8 files changed, 279 insertions(+), 91 deletions(-)
Changes applied before test
commit 808ae6851faee9b633e773f9150d360cdb927146
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:29:45 2021 +0100

    Allow overriding the timestamp of grab_next_visits
    
    This makes the simulator behavior more consistent with reality.

commit 9943195d31c51a44325cba09d07fb6e904d45a00
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:27:40 2021 +0100

    Construct grab_next_visits query arguments incrementally

commit 72070b7bf628788b6872e90a3f8ac8f0c01b70d9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:57:42 2021 +0100

    simulator: add simple lister simulation

commit 1f1aad459c4b0740ecbe96e9809e4b31f66bf999
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:54:53 2021 +0100

    Factor out ListedOrigin generation to use the OriginModel
    
    This generates consistent last_update values according to the model and
    simulated time.

commit b93aa5be2c2d5dc2130e1027698f3e1255052d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/233/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/233/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 21 2021, 6:13 PM
Harbormaster failed remote builds in B18624: Diff 17490!
swh-public-ci added a comment.Jan 22 2021, 11:29 AM

Build is green

Patch application report for D4911 (id=17490)

Could not rebase; Attempt merge onto 03460207a1...

Updating 0346020..808ae68
Fast-forward
 swh/scheduler/backend.py                    | 90 ++++++++++++++++++++++------
 swh/scheduler/interface.py                  | 45 ++++++++++----
 swh/scheduler/model.py                      | 33 +---------
 swh/scheduler/simulator/__init__.py         | 18 +++---
 swh/scheduler/simulator/origin_scheduler.py |  2 +-
 swh/scheduler/simulator/origins.py          | 83 ++++++++++++++++++++++++-
 swh/scheduler/tests/test_scheduler.py       | 93 +++++++++++++++++++++++++----
 swh/scheduler/tests/test_simulator.py       |  6 +-
 8 files changed, 279 insertions(+), 91 deletions(-)
Changes applied before test
commit 808ae6851faee9b633e773f9150d360cdb927146
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:29:45 2021 +0100

    Allow overriding the timestamp of grab_next_visits
    
    This makes the simulator behavior more consistent with reality.

commit 9943195d31c51a44325cba09d07fb6e904d45a00
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:27:40 2021 +0100

    Construct grab_next_visits query arguments incrementally

commit 72070b7bf628788b6872e90a3f8ac8f0c01b70d9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:57:42 2021 +0100

    simulator: add simple lister simulation

commit 1f1aad459c4b0740ecbe96e9809e4b31f66bf999
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:54:53 2021 +0100

    Factor out ListedOrigin generation to use the OriginModel
    
    This generates consistent last_update values according to the model and
    simulated time.

commit b93aa5be2c2d5dc2130e1027698f3e1255052d8d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 13:01:53 2021 +0100

    Make PaginatedListedOriginList a concretization of PagedResult
    
    1. consistent with swh-storage and swh-indexer-storage
    2. we can use swh.core.api.classes.stream_results on scheduler.get_listed_origins.

commit 2f47936731cf438a5195978a2af3250597b693b5
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:29:16 2021 +0100

    Add scheduling policy for already visited origins with known last update
    
    This policy schedules origins by decreasing order of "visit lag" (that
    is, origins with the most lag are scheduled first).

commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/251/ for more details.

Harbormaster completed remote builds in B18624: Diff 17490.Jan 22 2021, 11:29 AM
vlorentz requested review of this revision.Jan 22 2021, 11:29 AM
douardda accepted this revision.Jan 22 2021, 3:27 PM
This revision is now accepted and ready to land.Jan 22 2021, 3:27 PM
vlorentz updated this revision to Diff 17569.Jan 25 2021, 2:39 PM

rebase

swh-public-ci added a comment.Jan 25 2021, 2:46 PM

Build is green

Patch application report for D4911 (id=17569)

Could not rebase; Attempt merge onto 2906b4e8a0...

Updating 2906b4e..09a8768
Fast-forward
 swh/scheduler/backend.py                    | 44 +++++++++++---
 swh/scheduler/interface.py                  | 15 ++++-
 swh/scheduler/simulator/__init__.py         | 18 +++---
 swh/scheduler/simulator/origin_scheduler.py |  2 +-
 swh/scheduler/simulator/origins.py          | 92 +++++++++++++++++++++++++++--
 5 files changed, 144 insertions(+), 27 deletions(-)
Changes applied before test
commit 09a8768c30dc335afccde4df046b371a274cb2f9
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:29:45 2021 +0100

    Allow overriding the timestamp of grab_next_visits
    
    This makes the simulator behavior more consistent with reality.

commit a2dc72474056c2f20e255acf13ec3e662e1aad7a
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:27:40 2021 +0100

    Construct grab_next_visits query arguments incrementally

commit e5709214b4917a5fe3634d040da7a061f5978f66
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:57:42 2021 +0100

    simulator: add simple lister simulation

commit 7af98e2bc048c6946679e7d95cf8620e4a0ee4bf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:54:53 2021 +0100

    Factor out ListedOrigin generation to use the OriginModel
    
    This generates consistent last_update values according to the model and
    simulated time.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/273/ for more details.

Harbormaster completed remote builds in B18704: Diff 17569.Jan 25 2021, 2:46 PM
vlorentz updated this revision to Diff 17609.Jan 26 2021, 1:23 PM

rebase

swh-public-ci added a comment.Jan 26 2021, 1:31 PM

Build is green

Patch application report for D4911 (id=17609)

Could not rebase; Attempt merge onto 2906b4e8a0...

Updating 2906b4e..2b39cbc
Fast-forward
 docs/simulator.rst                          | 15 +++++
 swh/scheduler/backend.py                    | 44 +++++++++++---
 swh/scheduler/interface.py                  | 15 ++++-
 swh/scheduler/simulator/__init__.py         | 18 +++---
 swh/scheduler/simulator/origin_scheduler.py |  2 +-
 swh/scheduler/simulator/origins.py          | 92 +++++++++++++++++++++++++++--
 6 files changed, 159 insertions(+), 27 deletions(-)
Changes applied before test
commit 2b39cbcabf9960c1f660442e15f6c17654aec9e2
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:29:45 2021 +0100

    Allow overriding the timestamp of grab_next_visits
    
    This makes the simulator behavior more consistent with reality.

commit 7ffbdd1b3eb579f43e8913ea11cfd916b2f3c457
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Jan 21 17:27:40 2021 +0100

    Construct grab_next_visits query arguments incrementally

commit ea068b46a89e07c60ad1233afd36afc6bb29031e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:57:42 2021 +0100

    simulator: add simple lister simulation

commit 7af98e2bc048c6946679e7d95cf8620e4a0ee4bf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Jan 21 14:54:53 2021 +0100

    Factor out ListedOrigin generation to use the OriginModel
    
    This generates consistent last_update values according to the model and
    simulated time.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/282/ for more details.

Harbormaster completed remote builds in B18747: Diff 17609.Jan 26 2021, 1:31 PM
Closed by commit rDSCH2b39cbcabf99: Allow overriding the timestamp of grab_next_visits (authored by olasd, committed by vlorentz). · Explain WhyJan 29 2021, 10:00 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSCH2b39cbcabf99: Allow overriding the timestamp of grab_next_visits.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
scheduler/
23 lines91%
Loading...
15 lines69%
Loading...
simulator/
2 lines100%
Loading...

Diff 17609

View Options

swh/scheduler/backend.py

Loading...
View Options

swh/scheduler/interface.py

Loading...
View Options

swh/scheduler/simulator/origin_scheduler.py

Loading...
About · Developer information · Legal