Page MenuHomeSoftware Heritage
Differential D4846

Introduce a `swh scheduler origin schedule-next` cli
ClosedPublic
Actions

Authored by vlorentz on Jan 12 2021, 5:12 PM.

Details

Reviewers
olasd
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T2444: Implement the scheduling policy for the recurrent visit scheduler
Commits
rDSCH4f5338f2aba3: Introduce a `swh scheduler origin schedule-next` cli
Summary

This creates one-shot tasks in the classic scheduler for the next visits
to run according to the visit scheduling policy.

Depends on D4844

Test Plan

tests added

Diff Detail

Event Timeline

vlorentz created this revision.Jan 12 2021, 5:12 PM
swh-public-ci added a comment.Jan 12 2021, 5:14 PM

Build has FAILED

Patch application report for D4846 (id=17154)

Could not rebase; Attempt merge onto b13cb1f2de...

Updating b13cb1f..3ddec04
Fast-forward
 swh/scheduler/cli/__init__.py          |   7 +-
 swh/scheduler/cli/origin.py            | 135 +++++++++++++++++++++++++++++++++
 swh/scheduler/model.py                 |  11 ++-
 swh/scheduler/tests/test_cli_origin.py | 103 +++++++++++++++++++++++++
 swh/scheduler/tests/test_model.py      |  31 +++++++-
 5 files changed, 283 insertions(+), 4 deletions(-)
 create mode 100644 swh/scheduler/cli/origin.py
 create mode 100644 swh/scheduler/tests/test_cli_origin.py
Changes applied before test
commit 3ddec0454da1fb51f00f6c0d03e55ad22ca173eb
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:10:39 2021 +0100

    Introduce a `swh scheduler origin schedule-next` cli
    
    This creates one-shot tasks in the classic scheduler for the next visits
    to run according to the visit scheduling policy.

commit 3c15a60589efdb5268acbe9934e32a5cac2f47a0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 16:16:31 2021 +0100

    Introduce a `swh scheduler origin grab-next` cli
    
    This returns, as CSV, the next origins to be visited according to the
    passed scheduling policy.

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/97/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/97/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 12 2021, 5:14 PM
Harbormaster failed remote builds in B18298: Diff 17154!
vlorentz updated this revision to Diff 17156.Jan 12 2021, 5:32 PM

Rebase

swh-public-ci added a comment.Jan 12 2021, 5:35 PM

Build is green

Patch application report for D4846 (id=17156)

Could not rebase; Attempt merge onto b13cb1f2de...

Updating b13cb1f..f02b167
Fast-forward
 swh/scheduler/cli/__init__.py          |   7 +-
 swh/scheduler/cli/origin.py            | 135 +++++++++++++++++++++++++++++++++
 swh/scheduler/model.py                 |  11 ++-
 swh/scheduler/tests/common.py          |  10 +--
 swh/scheduler/tests/test_cli_origin.py | 103 +++++++++++++++++++++++++
 swh/scheduler/tests/test_model.py      |  31 +++++++-
 6 files changed, 288 insertions(+), 9 deletions(-)
 create mode 100644 swh/scheduler/cli/origin.py
 create mode 100644 swh/scheduler/tests/test_cli_origin.py
Changes applied before test
commit f02b16735073cae4cd512751ed007de6d253603e
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:10:39 2021 +0100

    Introduce a `swh scheduler origin schedule-next` cli
    
    This creates one-shot tasks in the classic scheduler for the next visits
    to run according to the visit scheduling policy.

commit 2560e71f5b0042b5d508c8ae3d6d580d7098620c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:28:33 2021 +0100

    Rename test task types to names that match real tasks
    
    The success of tests using these task types would depend on the test run
    order, because these task types are (currently) being created by
    swh/scheduler/sql/50-data.sql, but the table is truncated after the
    first test completes.

commit 3c15a60589efdb5268acbe9934e32a5cac2f47a0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 16:16:31 2021 +0100

    Introduce a `swh scheduler origin grab-next` cli
    
    This returns, as CSV, the next origins to be visited according to the
    passed scheduling policy.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/99/ for more details.

Harbormaster completed remote builds in B18300: Diff 17156.Jan 12 2021, 5:35 PM
vlorentz requested review of this revision.Jan 12 2021, 5:35 PM
ardumont accepted this revision.Jan 13 2021, 3:11 PM
This revision is now accepted and ready to land.Jan 13 2021, 3:11 PM
vlorentz updated this revision to Diff 17184.Jan 13 2021, 3:43 PM

rebase

vlorentz added a child revision: D4855: Filter origins by visit type when scheduling the next visits.Jan 13 2021, 3:44 PM
swh-public-ci added a comment.Jan 13 2021, 3:48 PM

Build is green

Patch application report for D4846 (id=17184)

Could not rebase; Attempt merge onto a62003397d...

Updating a620033..da347f7
Fast-forward
 swh/scheduler/cli/__init__.py          |   7 +-
 swh/scheduler/cli/origin.py            | 135 +++++++++++++++++++++++++++++++++
 swh/scheduler/model.py                 |   9 +++
 swh/scheduler/tests/common.py          |  10 +--
 swh/scheduler/tests/test_cli_origin.py | 103 +++++++++++++++++++++++++
 swh/scheduler/tests/test_model.py      |  31 +++++++-
 6 files changed, 287 insertions(+), 8 deletions(-)
 create mode 100644 swh/scheduler/cli/origin.py
 create mode 100644 swh/scheduler/tests/test_cli_origin.py
Changes applied before test
commit da347f7f4c401a43ec34de76365ad323d0ff7b77
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:10:39 2021 +0100

    Introduce a `swh scheduler origin schedule-next` cli
    
    This creates one-shot tasks in the classic scheduler for the next visits
    to run according to the visit scheduling policy.

commit 42957c9e96e6c7d8070e0b6c786c273e8c1602a0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:28:33 2021 +0100

    Rename test task types to names that match real tasks
    
    The success of tests using these task types would depend on the test run
    order, because these task types are (currently) being created by
    swh/scheduler/sql/50-data.sql, but the table is truncated after the
    first test completes.

commit d1393c54da99c45175dd0b6a69734d17fc887960
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 16:16:31 2021 +0100

    Introduce a `swh scheduler origin grab-next` cli
    
    This returns, as CSV, the next origins to be visited according to the
    passed scheduling policy.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/105/ for more details.

Harbormaster completed remote builds in B18328: Diff 17184.Jan 13 2021, 3:48 PM
vlorentz updated this revision to Diff 17214.Jan 14 2021, 1:55 PM

rebase

swh-public-ci added a comment.Jan 14 2021, 2:00 PM

Build is green

Patch application report for D4846 (id=17214)

Could not rebase; Attempt merge onto a62003397d...

Updating a620033..4f5338f
Fast-forward
 swh/scheduler/cli/__init__.py          |   7 +-
 swh/scheduler/cli/origin.py            | 136 +++++++++++++++++++++++++++++++++
 swh/scheduler/model.py                 |   9 +++
 swh/scheduler/tests/common.py          |  10 +--
 swh/scheduler/tests/test_cli_origin.py | 103 +++++++++++++++++++++++++
 swh/scheduler/tests/test_model.py      |  31 +++++++-
 6 files changed, 288 insertions(+), 8 deletions(-)
 create mode 100644 swh/scheduler/cli/origin.py
 create mode 100644 swh/scheduler/tests/test_cli_origin.py
Changes applied before test
commit 4f5338f2aba360fed2e524cbcdd23b11bacfb79d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:10:39 2021 +0100

    Introduce a `swh scheduler origin schedule-next` cli
    
    This creates one-shot tasks in the classic scheduler for the next visits
    to run according to the visit scheduling policy.

commit 3dd1d5f28d329620a65ee00749d24401b6d8cf00
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:28:33 2021 +0100

    Rename test task types to names that match real tasks
    
    The success of tests using these task types would depend on the test run
    order, because these task types are (currently) being created by
    swh/scheduler/sql/50-data.sql, but the table is truncated after the
    first test completes.

commit 5d7b002ac403565e348ac8fe4dd56d015cf29cae
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 16:16:31 2021 +0100

    Introduce a `swh scheduler origin grab-next` cli
    
    This returns, as CSV, the next origins to be visited according to the
    passed scheduling policy.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/118/ for more details.

Harbormaster completed remote builds in B18358: Diff 17214.Jan 14 2021, 2:00 PM
Closed by commit rDSCH4f5338f2aba3: Introduce a `swh scheduler origin schedule-next` cli (authored by olasd, committed by vlorentz). · Explain WhyJan 14 2021, 2:16 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSCH4f5338f2aba3: Introduce a `swh scheduler origin schedule-next` cli.
olasd added a task: T2444: Implement the scheduling policy for the recurrent visit scheduler.Jan 18 2021, 2:14 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
scheduler/
cli/
34 lines95%
Loading...
9 lines98%
Loading...
tests/
27 lines100%
Loading...
31 lines96%
Loading...

Diff 17184

View Options

swh/scheduler/cli/origin.py

Loading...
View Options

swh/scheduler/model.py

Loading...
View Options

swh/scheduler/tests/test_cli_origin.py

Loading...
View Options

swh/scheduler/tests/test_model.py

Loading...
About · Developer information · Legal