Page MenuHomeSoftware Heritage

Add scheduling policy for never visited origins
ClosedPublic

Authored by vlorentz on Jan 20 2021, 5:46 PM.

Details

Summary

This policy orders never visited origins by increasing date of last
update (scheduling the "oldest" never visited origins first).

Related to T2444

Event Timeline

Build is green

Patch application report for D4898 (id=17410)

Could not rebase; Attempt merge onto 898820fac5...

Updating 898820f..a5ece5f
Fast-forward
 swh/scheduler/backend.py              |  55 +++++++++++-----
 swh/scheduler/tests/test_scheduler.py | 121 +++++++++++++++++++++++++++-------
 2 files changed, 134 insertions(+), 42 deletions(-)
Changes applied before test
commit a5ece5f6aa7a378062eeb6ab8e7c9b0faea35c11
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit e03158823653265bd5ebcb60d7bcc67c0e8beb4e
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

commit 8bab1ba37aebbb9921e73ffbb17a9cb25a94c264
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:17:17 2021 +0100

    Make the grab_next_visits sql query modular
    
    This will allow us to easily plug new scheduling policies in that
    function.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/197/ for more details.

Build is green

Patch application report for D4898 (id=17459)

Could not rebase; Attempt merge onto b641ac83eb...

Updating b641ac8..ae71389
Fast-forward
 swh/scheduler/backend.py              |   7 ++
 swh/scheduler/tests/test_scheduler.py | 130 ++++++++++++++++++++++++++--------
 2 files changed, 109 insertions(+), 28 deletions(-)
Changes applied before test
commit ae71389ef603c6454d18e180c6c9019f99dcda9d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 6ea71f094c81df4d60a5d5872e4b60c2b4dc0f7c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/215/ for more details.

Build has FAILED

Patch application report for D4898 (id=17471)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..acad712
Fast-forward
 swh/scheduler/backend.py              |   7 ++
 swh/scheduler/tests/test_scheduler.py | 130 ++++++++++++++++++++++++++--------
 2 files changed, 109 insertions(+), 28 deletions(-)
Changes applied before test
commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/219/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/219/console

Build has FAILED

Patch application report for D4898 (id=17471)

Could not rebase; Attempt merge onto af3789891f...

Updating af37898..acad712
Fast-forward
 swh/scheduler/backend.py              |   7 ++
 swh/scheduler/tests/test_scheduler.py | 130 ++++++++++++++++++++++++++--------
 2 files changed, 109 insertions(+), 28 deletions(-)
Changes applied before test
commit acad712ad3f71f88f99e45e9b4f571ad751945dc
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 03460207a17d82635ef5a6f12358392143eb9eef
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/222/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/222/console

Build is green

Patch application report for D4898 (id=17459)

Could not rebase; Attempt merge onto af3789891f...

Merge made by the 'recursive' strategy.
 swh/scheduler/backend.py              |   7 ++
 swh/scheduler/tests/test_scheduler.py | 130 ++++++++++++++++++++++++++--------
 2 files changed, 109 insertions(+), 28 deletions(-)
Changes applied before test
commit 21d52ff17b70bf04897581964198d6271b4cde7e
Merge: af37898 ae71389
Author: Jenkins user <jenkins@localhost>
Date:   Thu Jan 21 12:39:04 2021 +0000

    Merge branch 'diff-target' into HEAD

commit ae71389ef603c6454d18e180c6c9019f99dcda9d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:25:46 2021 +0100

    Add scheduling policy for never visited origins
    
    This policy orders never visited origins by increasing date of last
    update (scheduling the "oldest" never visited origins first).

commit 6ea71f094c81df4d60a5d5872e4b60c2b4dc0f7c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 20 17:23:03 2021 +0100

    Reorganize grab_next_visits tests to better check sorting behavior
    
     - factor out test setup and results checking
     - properly exercize corner cases of the oldest_scheduled_first policy

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/226/ for more details.

This revision is now accepted and ready to land.Jan 21 2021, 7:30 PM
This revision was landed with ongoing or failed builds.Jan 22 2021, 11:34 AM
This revision was automatically updated to reflect the committed changes.