Page MenuHomeSoftware Heritage

scheduler: Redirect priority task in their own dedicated task queue
AbandonedPublicDraft

Authored by ardumont on Apr 13 2021, 1:22 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T3084: Fast track save code now requests
Summary

This simply persists tasks with priority in another task type. And let the scheduler
runner and listener deals with them as other task types does (avoid some otherwise not
so fancy computations [2]).

This implies that the dedicated task type exists in the scheduler backend [1].

Instead of doing some not so fancy computations in the scheduler runner. Simply
detect and persist task into a dedicated high task type.

That way, future save code now requests will be in their own queue and consumed by
dedicated workers [3]. Existing save code now requests will eventually be consumed as it is
today (if any remained not done yet).

Related to T3084

[1] D5488

[2] D5488#139373 (described the detail of what that would entail otherwise)

[3] D5486

Test Plan

tox
(failing as i've yet to understand what's going wrong in there, fixture
override is not used somehow for starter...)

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 20664
Build 32066: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 32065: arc lint + arc unit

Event Timeline

Build has FAILED

Patch application report for D5493 (id=19636)

Rebasing onto ecab745a5f...

Current branch diff-target is up to date.
Changes applied before test
commit 0bd77909f8a48066fcb51e90ecaaacba1e0a7764
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Apr 13 13:13:56 2021 +0200

    scheduler: Redirect priority task in their own dedicated task queue
    
    This implies that dedicated task type exists in the scheduler backend.
    
    Related to T3084

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/304/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/304/console

Harbormaster returned this revision to the author for changes because remote builds failed.Apr 13 2021, 1:23 PM
Harbormaster failed remote builds in B20664: Diff 19636!
ardumont added inline comments.
swh/scheduler/tests/conftest.py
31

fixture unused!!!???

swh/scheduler/tests/conftest.py
31

bypassed due to [1] which is not clear what that means or does...
I recall it's about speeding tests but still...

[1] https://forge.softwareheritage.org/source/swh-scheduler/browse/master/swh/scheduler/tests/test_api_client.py$0-31

After discussion with @olasd, we'll simplify the scheduler runner instead:

  • split the reading methods, 2 for reading standard tasks (peek,grab), another 2 to read the ones with priority
  • orchestrate the reading of priority tasks in the runner
  • leave the existing behavior for standard messages for now
  • Deploy

No need to create dedicated tasks in the respective loaders.

And later, we'll be able to simplify the priority ratio stanza in the scheduler runner (drop drop drop the dead code!)

superseded according to the earlier plan ^ and by the work started in D5503