Paths

Table of Contentst

Differential D1196

Add an 'swh-scheduler task schedule_origins' command to run a task on all origins.
ClosedPublic
Actions

Authored by vlorentz on Feb 26 2019, 2:21 PM.

Tags

None

Subscribers

Details

Reviewers

Group Reviewers

Maniphest Tasks

T1536: Add a CLI tool to run a task on all origins

Commits

rDSCH763dbda4b652: Ignore INFO messages when testing CLI output.
rDSCH1122ee12e067: Add a 'task schedule_origins' command to run a task on all origins.

Summary

Depends on D1195
Resolves T1536

Note: schedule_origin_batches is copied verbatim from swh-indexer, I'm going to open a new diff in the indexer to deduplicate this code.

Diff Detail

Repository

rDSCH Scheduling utilities

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Feb 26 2019, 2:21 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 26 2019, 2:21 PM

Harbormaster failed remote builds in B4453: Diff 3761!Feb 26 2019, 2:21 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/205/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/205/console

vlorentz edited the summary of this revision. (Show Details)Feb 26 2019, 2:22 PM

vlorentz added a child revision: D1197: Deduplicate origin batch scheduling wrt the scheduler..Feb 26 2019, 2:24 PM

vlorentz added a parent revision: D1193: Add an helper function to list all origins in the storage..Mar 1 2019, 2:25 PM

douardda requested changes to this revision.Mar 8 2019, 11:52 AM

douardda added a subscriber: douardda.

douardda added inline comments.

swh/scheduler/cli.py
350	nitpicking: the --no-dry-run is useless, IMHO. But it's already like this in other subcommands, so meh.
357	Please add a bit more doc in there. What value is expected for `type`? What are `options` arguments for? Maybe an example might be handy?
swh/scheduler/cli_utils.py
72	itertools.islice?
swh/scheduler/tests/test_cli.py
613	maybe strip the logging statements from the expectations in the test? I really don't like a test to fail because logging has been added/removed/modifed.

This revision now requires changes to proceed.Mar 8 2019, 11:52 AM

Rebase
Add a 'task schedule_origins' command to run a task on all origins.
Document arguments 'type' and 'options'.
itertools
Document schedule_task and schedule_origin_metadata_index.

fix Diff base

vlorentz marked 4 inline comments as done.Mar 11 2019, 2:29 PM

vlorentz added inline comments.

swh/scheduler/cli_utils.py
72	I keep forgetting about this function! Thanks
swh/scheduler/tests/test_cli.py
613	I check against it to make sure there is no other output before `Scheduled 3 tasks \(30 origins\).` . I agree that's not ideal, but it's better than a silent error/warning.

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/221/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/221/console

Harbormaster failed remote builds in B4590: Diff 3884!Mar 11 2019, 2:29 PM

Harbormaster failed remote builds in B4591: Diff 3885!

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/222/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/222/console

rebase

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/226/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/226/console

Harbormaster failed remote builds in B4625: Diff 3911!Mar 13 2019, 2:43 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/227/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/227/console

fix typo

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/228/ for more details.

Harbormaster completed remote builds in B4626: Diff 3912.Mar 13 2019, 3:25 PM

typo + I really think this logging statements should not interfere with the test.

swh/scheduler/cli.py
357	known?
swh/scheduler/tests/test_cli.py
613	I was not suggesting to silence them, just to add a regex to ignore them when asserting

This revision now requires changes to proceed.Mar 14 2019, 11:31 AM

fix typo
Ignore INFO messages when testing CLI output.

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/230/ for more details.

Harbormaster completed remote builds in B4635: Diff 3923.Mar 14 2019, 1:20 PM

douardda accepted this revision.Mar 14 2019, 4:32 PM

This revision is now accepted and ready to land.Mar 14 2019, 4:32 PM

rebase

Closed by commit rDSCH1122ee12e067: Add a 'task schedule_origins' command to run a task on all origins. (authored by vlorentz). · Explain WhyMar 14 2019, 4:36 PM

This revision was automatically updated to reflect the committed changes.

Harbormaster failed remote builds in B4641: Diff 3929!Mar 14 2019, 4:36 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/231/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/231/console

Revision Contents
Changeset List

Path

Size

requirements-swh.txt

1 line

swh/

scheduler/

57 lines

42 lines

tests/

106 lines

Diff 3930

requirements-swh.txt

Loading...

swh/scheduler/cli.py

Loading...

swh/scheduler/cli_utils.py

Loading...

swh/scheduler/tests/test_cli.py

Loading...