Page MenuHomeSoftware Heritage
Differential D4855

Filter origins by visit type when scheduling the next visits
ClosedPublic
Actions

Authored by vlorentz on Jan 13 2021, 3:44 PM.

Details

Reviewers
olasd
ardumont
Group Reviewers
Reviewers
Commits
rDSCHca45d40f2a62: Filter origins by visit type when scheduling the next visits
Summary

We have separate task queues and workers for each visit type, so it
makes sense to split this endpoint along these lines too, at least for
now.

Depends on D4846
Depends on D4854

Test Plan

updated tests for the new arguments

Diff Detail

Repository
rDSCH Scheduling utilities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jan 13 2021, 3:44 PM
swh-public-ci added a comment.Jan 13 2021, 3:52 PM

Build is green

Patch application report for D4855 (id=17186)

Could not rebase; Attempt merge onto a62003397d...

Updating a620033..9f843ee
Fast-forward
 sql/updates/20.sql                     |   6 ++
 swh/scheduler/backend.py               |   9 ++-
 swh/scheduler/cli/__init__.py          |   7 +-
 swh/scheduler/cli/origin.py            | 141 +++++++++++++++++++++++++++++++++
 swh/scheduler/interface.py             |   8 +-
 swh/scheduler/model.py                 |   9 +++
 swh/scheduler/sql/30-schema.sql        |   2 +-
 swh/scheduler/sql/60-indexes.sql       |   2 +-
 swh/scheduler/tests/common.py          |  10 +--
 swh/scheduler/tests/conftest.py        |  45 ++++++++---
 swh/scheduler/tests/test_cli_origin.py | 112 ++++++++++++++++++++++++++
 swh/scheduler/tests/test_model.py      |  31 +++++++-
 swh/scheduler/tests/test_scheduler.py  |  27 ++++---
 13 files changed, 371 insertions(+), 38 deletions(-)
 create mode 100644 sql/updates/20.sql
 create mode 100644 swh/scheduler/cli/origin.py
 create mode 100644 swh/scheduler/tests/test_cli_origin.py
Changes applied before test
commit 9f843eef37313b551a158dfa11aea97e5ef2fc81
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 13 15:31:55 2021 +0100

    Filter origins by visit type when scheduling the next visits
    
    We have separate task queues and workers for each visit type, so it
    makes sense to split this endpoint along these lines too, at least for
    now.

commit 23d1b3c1883c3c955b5dd5ba1cc2270c93e156d6
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 13 15:25:56 2021 +0100

    Reorganize ListedOrigin fixtures to generate multiple visit_types

commit da347f7f4c401a43ec34de76365ad323d0ff7b77
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:10:39 2021 +0100

    Introduce a `swh scheduler origin schedule-next` cli
    
    This creates one-shot tasks in the classic scheduler for the next visits
    to run according to the visit scheduling policy.

commit 42957c9e96e6c7d8070e0b6c786c273e8c1602a0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:28:33 2021 +0100

    Rename test task types to names that match real tasks
    
    The success of tests using these task types would depend on the test run
    order, because these task types are (currently) being created by
    swh/scheduler/sql/50-data.sql, but the table is truncated after the
    first test completes.

commit d1393c54da99c45175dd0b6a69734d17fc887960
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 16:16:31 2021 +0100

    Introduce a `swh scheduler origin grab-next` cli
    
    This returns, as CSV, the next origins to be visited according to the
    passed scheduling policy.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/107/ for more details.

Harbormaster completed remote builds in B18330: Diff 17186.Jan 13 2021, 3:52 PM
vlorentz requested review of this revision.Jan 13 2021, 3:52 PM
ardumont mentioned this in D4853: visit_stats: Keep a reference to the last snapshot.Jan 13 2021, 6:47 PM
ardumont accepted this revision.Jan 13 2021, 6:56 PM
This revision is now accepted and ready to land.Jan 13 2021, 6:56 PM
vlorentz updated this revision to Diff 17216.Jan 14 2021, 1:55 PM

rebase

swh-public-ci added a comment.Jan 14 2021, 2:04 PM

Build is green

Patch application report for D4855 (id=17216)

Could not rebase; Attempt merge onto a62003397d...

Updating a620033..ca45d40
Fast-forward
 sql/updates/20.sql                     |   6 ++
 swh/scheduler/backend.py               |   9 ++-
 swh/scheduler/cli/__init__.py          |   7 +-
 swh/scheduler/cli/origin.py            | 142 +++++++++++++++++++++++++++++++++
 swh/scheduler/interface.py             |   8 +-
 swh/scheduler/model.py                 |   9 +++
 swh/scheduler/sql/30-schema.sql        |   2 +-
 swh/scheduler/sql/60-indexes.sql       |   2 +-
 swh/scheduler/tests/common.py          |  10 +--
 swh/scheduler/tests/conftest.py        |  45 ++++++++---
 swh/scheduler/tests/test_cli_origin.py | 112 ++++++++++++++++++++++++++
 swh/scheduler/tests/test_model.py      |  31 ++++++-
 swh/scheduler/tests/test_scheduler.py  |  27 ++++---
 13 files changed, 372 insertions(+), 38 deletions(-)
 create mode 100644 sql/updates/20.sql
 create mode 100644 swh/scheduler/cli/origin.py
 create mode 100644 swh/scheduler/tests/test_cli_origin.py
Changes applied before test
commit ca45d40f2a62d4a0f200cabe760ad3a0cda00f89
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 13 15:31:55 2021 +0100

    Filter origins by visit type when scheduling the next visits
    
    We have separate task queues and workers for each visit type, so it
    makes sense to split this endpoint along these lines too, at least for
    now.

commit 59b4cb3f1c7a081e0d28b11d15888d38a9de151e
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Jan 13 15:25:56 2021 +0100

    Reorganize ListedOrigin fixtures to generate multiple visit_types

commit 4f5338f2aba360fed2e524cbcdd23b11bacfb79d
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:10:39 2021 +0100

    Introduce a `swh scheduler origin schedule-next` cli
    
    This creates one-shot tasks in the classic scheduler for the next visits
    to run according to the visit scheduling policy.

commit 3dd1d5f28d329620a65ee00749d24401b6d8cf00
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 17:28:33 2021 +0100

    Rename test task types to names that match real tasks
    
    The success of tests using these task types would depend on the test run
    order, because these task types are (currently) being created by
    swh/scheduler/sql/50-data.sql, but the table is truncated after the
    first test completes.

commit 5d7b002ac403565e348ac8fe4dd56d015cf29cae
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue Jan 12 16:16:31 2021 +0100

    Introduce a `swh scheduler origin grab-next` cli
    
    This returns, as CSV, the next origins to be visited according to the
    passed scheduling policy.

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/120/ for more details.

Harbormaster completed remote builds in B18360: Diff 17216.Jan 14 2021, 2:04 PM
Closed by commit rDSCHca45d40f2a62: Filter origins by visit type when scheduling the next visits (authored by olasd, committed by vlorentz). · Explain WhyJan 14 2021, 2:16 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSCHca45d40f2a62: Filter origins by visit type when scheduling the next visits.

Revision Contents
Changeset List

PathSize
sql/
updates/
6 lines
swh/
scheduler/
9 lines
cli/
18 lines
8 lines
sql/
2 lines
2 lines
tests/
37 lines
27 lines

Diff 17223

View Options

sql/updates/20.sql

Loading...
View Options

swh/scheduler/backend.py

Loading...
View Options

swh/scheduler/cli/origin.py

Loading...
View Options

swh/scheduler/interface.py

Loading...
View Options

swh/scheduler/sql/30-schema.sql

Loading...
View Options

swh/scheduler/sql/60-indexes.sql

Loading...
View Options

swh/scheduler/tests/test_cli_origin.py

Loading...
View Options

swh/scheduler/tests/test_scheduler.py

Loading...
About · Developer information · Legal