Page MenuHomeSoftware Heritage
Differential D605

Add SchedulerTestFixture.
ClosedPublic
Actions

Authored by vlorentz on Oct 25 2018, 7:06 PM.

Details

Reviewers
douardda
ardumont
Group Reviewers
Reviewers
Commits
rDSCH762b5283e1cc: Add SchedulerTestFixture.
Summary

Will be used by the indexers.

Test Plan

There's a test included

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
SchedulerTestFixture
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 1948
Build 2347: tox-on-jenkinsJenkins
Build 2346: arc lint + arc unit

Event Timeline

vlorentz created this revision.Oct 25 2018, 7:06 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 25 2018, 7:06 PM
swh-public-ci added a subscriber: swh-public-ci.Oct 25 2018, 7:09 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/12/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/12/console

Harbormaster failed remote builds in B1934: Diff 1847!Oct 25 2018, 7:09 PM
vlorentz added a child revision: D606: Use swh.scheduler instead of celery in the orchestrator..Oct 26 2018, 7:21 AM
douardda requested changes to this revision.Oct 26 2018, 9:26 AM
douardda added a subscriber: douardda.
douardda added inline comments.
swh/scheduler/celery_backend/runner.py
38–68

There should be at least an explanation about why this refactoring in the celery runner is required to add a test fixture, and give a clue what's going one in this refactoring.

Besides, you de facto change the API of the run_ready_tasks function so this must be documented in the doc string and the rationals for this API change should be explained in the commit message.

This revision now requires changes to proceed.Oct 26 2018, 9:26 AM
vlorentz updated this revision to Diff 1856.Oct 26 2018, 9:41 AM
  • Make SchedulerTestFixture inherit from CeleryTestFixture.
swh-public-ci added a comment.Oct 26 2018, 9:43 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/13/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/13/console

Harbormaster failed remote builds in B1941: Diff 1856!Oct 26 2018, 9:43 AM
vlorentz added a child revision: D608: Use swh.scheduler instead of celery in the OriginHeadIndexer..Oct 26 2018, 10:43 AM
vlorentz marked an inline comment as done.Oct 26 2018, 10:47 AM
vlorentz updated this revision to Diff 1859.Oct 26 2018, 10:47 AM
  • Document the return value of run_ready_tasks.
swh-public-ci added a comment.Oct 26 2018, 10:49 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tox/14/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tox/14/console

Harbormaster failed remote builds in B1947: Diff 1859!Oct 26 2018, 10:49 AM
vlorentz updated this revision to Diff 1860.Oct 26 2018, 10:51 AM
  • Fix MRO.
  • Fix test name.
swh-public-ci added a comment.Oct 26 2018, 10:53 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/15/ for more details.

Harbormaster completed remote builds in B1948: Diff 1860.Oct 26 2018, 10:53 AM
ardumont added a subscriber: ardumont.Oct 26 2018, 11:42 AM
ardumont added inline comments.
swh/scheduler/celery_backend/runner.py
93

If I am understanding this correcly, that will become quite large... (as in too large, MemoryError)
We do always have pending tasks for at least the loader-git... [1]

[1] https://grafana.softwareheritage.org/d/LmGkNMNik/sofwareheritage-scheduler?orgId=1
task_type in question is 'origin-update-git'

ardumont added inline comments.Oct 26 2018, 11:49 AM
swh/scheduler/celery_backend/runner.py
93

Ah but as you pointed out, there is the boundary (max_queue_length and all that ;).
Cool, nevermind then ;)

vlorentz marked 2 inline comments as done.Oct 26 2018, 12:00 PM
ardumont added inline comments.Oct 26 2018, 1:39 PM
swh/scheduler/tests/test_fixtures.py
16

Why not use swh.scheduler.tasks here since that will reflect the actual reality of what we run?

ardumont added a comment.Oct 26 2018, 1:43 PM

Sounds good.

Tests fine through tox.

swh/scheduler/tests/test_fixtures.py
39

Using swh.scheduler.utils.create_task_dict?

That'd make one more entrypoint used by tests ;)

vlorentz updated this revision to Diff 1864.Oct 26 2018, 1:45 PM
  • Inherit from swh.scheduler.task.Task.
ardumont added a comment.Oct 26 2018, 1:46 PM

Remains 2 nitpicks to either discuss or adapt, as you wish ;)

swh-public-ci added a comment.Oct 26 2018, 1:48 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/16/ for more details.

Harbormaster completed remote builds in B1952: Diff 1864.Oct 26 2018, 1:48 PM
vlorentz marked 2 inline comments as done.Oct 26 2018, 1:50 PM
vlorentz updated this revision to Diff 1865.Oct 26 2018, 1:50 PM
  • Use swh.scheduler.utils.create_task_dict.
swh-public-ci added a comment.Oct 26 2018, 1:53 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/17/ for more details.

Harbormaster completed remote builds in B1953: Diff 1865.Oct 26 2018, 1:53 PM
ardumont accepted this revision.Oct 26 2018, 1:58 PM
ardumont added a comment.Oct 29 2018, 9:42 AM

jsyk, will need rebase on latest merged scheduler diffs.

vlorentz updated this revision to Diff 1881.Oct 29 2018, 9:44 AM
  • Rebase
vlorentz updated this revision to Diff 1882.Oct 29 2018, 9:45 AM
  • Add docstring.
douardda accepted this revision.Oct 29 2018, 9:45 AM
This revision is now accepted and ready to land.Oct 29 2018, 9:45 AM
swh-public-ci added a comment.Oct 29 2018, 9:48 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/26/ for more details.

Harbormaster completed remote builds in B1970: Diff 1881.Oct 29 2018, 9:48 AM
swh-public-ci added a comment.Oct 29 2018, 9:51 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSCH/job/tox/27/ for more details.

Harbormaster completed remote builds in B1971: Diff 1882.Oct 29 2018, 9:51 AM
Closed by commit rDSCH762b5283e1cc: Add SchedulerTestFixture. (authored by vlorentz). · Explain WhyOct 29 2018, 9:54 AM
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
scheduler/
celery_backend/
27 lines
tests/
69 lines
43 lines

Diff 1860

View Options

swh/scheduler/celery_backend/runner.py

Loading...
View Options

swh/scheduler/tests/scheduler_testing.py

Loading...
View Options

swh/scheduler/tests/test_fixtures.py

Loading...
About · Developer information · Legal