Page MenuHomeSoftware Heritage
Differential D4898

Add scheduling policy for never visited origins
ClosedPublic
Actions

Authored by vlorentz on Jan 20 2021, 5:46 PM.

Details

Reviewers
olasd
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T2444: Implement the scheduling policy for the recurrent visit scheduler
Commits
rDSCHacad712ad3f7: Add scheduling policy for never visited origins
Summary

This policy orders never visited origins by increasing date of last
update (scheduling the "oldest" never visited origins first).

Related to T2444

Diff Detail

Event Timeline

vlorentz created this revision.Jan 20 2021, 5:46 PM
swh-public-ci added a comment.Jan 20 2021, 5:53 PM

Build is green

Patch application report for D4898 (id=17410)

Could not rebase; Attempt merge onto 898820fac5...

Updating 898820f..a5ece5f
Fast-forward
 swh/scheduler/backend.py              |  55 +++++++++++-----
 swh/scheduler/tests/test_scheduler.py | 121 +++++++++++++++++++++++++++-------
 2 files changed, 134 insertions(+), 42 deletions(-)
Changes applied before test
commit a5ece5f6aa7a378062eeb6ab8e7c9b0faea35c11
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit e03158823653265bd5ebcb60d7bcc67c0e8beb4e
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

commit 8bab1ba37aebbb9921e73ffbb17a9cb25a94c264
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:17:17 2021 +0100

    Make the grab_next_visits sql query modular
    
    This will allow us to easily plug new scheduling policies in that
    function.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/197/ for more details.

Harbormaster completed remote builds in B18547: Diff 17410.Jan 20 2021, 5:53 PM
vlorentz requested review of this revision.Jan 20 2021, 5:53 PM
vlorentz updated this revision to Diff 17456.Jan 21 2021, 11:28 AM

rebase

Harbormaster failed remote builds in B18592: Diff 17456!Jan 21 2021, 11:30 AM
swh-public-ci added a comment.Jan 21 2021, 11:30 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/212/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/212/console

vlorentz updated this revision to Diff 17459.Jan 21 2021, 11:36 AM

rebase

swh-public-ci added a comment.Jan 21 2021, 11:43 AM

Build is green

Patch application report for D4898 (id=17459)

Could not rebase; Attempt merge onto b641ac83eb...

Updating b641ac8..ae71389
Fast-forward
 swh/scheduler/backend.py              |   7 ++
 swh/scheduler/tests/test_scheduler.py | 130 ++++++++++++++++++++++++++--------
 2 files changed, 109 insertions(+), 28 deletions(-)
Changes applied before test
commit ae71389ef603c6454d18e180c6c9019f99dcda9d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 6ea71f094c81df4d60a5d5872e4b60c2b4dc0f7c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/215/ for more details.

Harbormaster completed remote builds in B18595: Diff 17459.Jan 21 2021, 11:43 AM
vlorentz updated this revision to Diff 17471.Jan 21 2021, 1:03 PM

rebase

swh-public-ci added a comment.Jan 21 2021, 1:04 PM

Build has FAILED

Patch application report for D4898 (id=17471)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..acad712
Fast-forward
 swh/scheduler/backend.py              |   7 ++
 swh/scheduler/tests/test_scheduler.py | 130 ++++++++++++++++++++++++++--------
 2 files changed, 109 insertions(+), 28 deletions(-)
Changes applied before test
commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/219/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/219/console

Harbormaster failed remote builds in B18605: Diff 17471!Jan 21 2021, 1:04 PM
swh-public-ci added a comment.Jan 21 2021, 1:08 PM

Build has FAILED

Patch application report for D4898 (id=17471)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..acad712
Fast-forward
 swh/scheduler/backend.py              |   7 ++
 swh/scheduler/tests/test_scheduler.py | 130 ++++++++++++++++++++++++++--------
 2 files changed, 109 insertions(+), 28 deletions(-)
Changes applied before test
commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/222/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/222/console

swh-public-ci added a comment.Jan 21 2021, 1:41 PM

Build is green

Patch application report for D4898 (id=17459)

Could not rebase; Attempt merge onto af3789891f...

Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py              |   7 ++
 swh/scheduler/tests/test_scheduler.py | 130 ++++++++++++++++++++++++++--------
 2 files changed, 109 insertions(+), 28 deletions(-)
Changes applied before test
commit 21d52ff17b70bf04897581964198d6271b4cde7e
Merge: af37898 ae71389
Author: Jenkins user <jenkins@localhost>
Date:   Thu Jan 21 12:39:04 2021 +0000

    Merge branch 'diff-target' into HEAD

commit ae71389ef603c6454d18e180c6c9019f99dcda9d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 6ea71f094c81df4d60a5d5872e4b60c2b4dc0f7c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/226/ for more details.

ardumont accepted this revision.Jan 21 2021, 7:30 PM
This revision is now accepted and ready to land.Jan 21 2021, 7:30 PM
This revision was landed with ongoing or failed builds.Jan 22 2021, 11:34 AM
Closed by commit rDSCHacad712ad3f7: Add scheduling policy for never visited origins (authored by olasd, committed by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSCHacad712ad3f7: Add scheduling policy for never visited origins.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
scheduler/
7 lines91%
Loading...
tests/
26 lines99%
Loading...

Diff 17459

View Options

swh/scheduler/backend.py

Loading...
View Options

swh/scheduler/tests/test_scheduler.py

Loading...
About · Developer information · Legal