Page MenuHomeSoftware Heritage

Add SchedulerTestFixture.
ClosedPublic

Authored by vlorentz on Oct 25 2018, 7:06 PM.

Details

Summary

Will be used by the indexers.

Test Plan

There's a test included

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
SchedulerTestFixture
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 1948
Build 2347: tox-on-jenkinsJenkins
Build 2346: arc lint + arc unit

Event Timeline

douardda added a subscriber: douardda.
douardda added inline comments.
swh/scheduler/celery_backend/runner.py
38–68

There should be at least an explanation about why this refactoring in the celery runner is required to add a test fixture, and give a clue what's going one in this refactoring.

Besides, you de facto change the API of the run_ready_tasks function so this must be documented in the doc string and the rationals for this API change should be explained in the commit message.

This revision now requires changes to proceed.Oct 26 2018, 9:26 AM
  • Make SchedulerTestFixture inherit from CeleryTestFixture.
  • Document the return value of run_ready_tasks.
ardumont added inline comments.
swh/scheduler/celery_backend/runner.py
93

If I am understanding this correcly, that will become quite large... (as in too large, MemoryError)
We do always have pending tasks for at least the loader-git... [1]

[1] https://grafana.softwareheritage.org/d/LmGkNMNik/sofwareheritage-scheduler?orgId=1
task_type in question is 'origin-update-git'

swh/scheduler/celery_backend/runner.py
93

Ah but as you pointed out, there is the boundary (max_queue_length and all that ;).
Cool, nevermind then ;)

swh/scheduler/tests/test_fixtures.py
16

Why not use swh.scheduler.tasks here since that will reflect the actual reality of what we run?

Sounds good.

Tests fine through tox.

swh/scheduler/tests/test_fixtures.py
39

Using swh.scheduler.utils.create_task_dict?

That'd make one more entrypoint used by tests ;)

  • Inherit from swh.scheduler.task.Task.

Remains 2 nitpicks to either discuss or adapt, as you wish ;)

  • Use swh.scheduler.utils.create_task_dict.

jsyk, will need rebase on latest merged scheduler diffs.

This revision is now accepted and ready to land.Oct 29 2018, 9:45 AM
This revision was automatically updated to reflect the committed changes.