Page MenuHomeSoftware Heritage
Create Task
Maniphest T3271

scheduler: Clean up dead code about priority/ratio
Closed, MigratedEdits Locked
Actions

Description

Following T3084, the priority code is no longer relevant.

Priority tasks are still used, they are rerouted by the runner to dedicated queues.
The number of tasks with a given ratio per priority is no longer used, that's the
part that needs clean up.

Clean it up from the base code.

Revisions and Commits

rDSCH Scheduling utilities
D5561rDSCHf33f74346f33 Simplify priority computation in tests + improve exhaustivity
D5552rDSCHbefccb94d608 scheduler: Clean up priority/ratio task dead code

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Apr 19 2021, 11:30 AM
ardumont created this task.
ardumont updated the task description. (Show Details)
ardumont added a revision: D5552: scheduler: Clean up priority/ratio task dead code.Apr 19 2021, 12:36 PM
ardumont added a commit: rDSCHbefccb94d608: scheduler: Clean up priority/ratio task dead code.Apr 20 2021, 11:45 AM
ardumont added a comment.EditedApr 20 2021, 12:18 PM

Deployed the new scheduler 0.13 version on staging.

But that seems to have slowed the scheduler runner down somehow.

Apr 20 10:02:22 scheduler0 swh[1499079]: INFO:swh.scheduler.celery_backend.runner:Grabbed 96 tasks index-origin-metadata
Apr 20 10:06:16 scheduler0 swh[1499079]: INFO:swh.scheduler.celery_backend.runner:Grabbed 1768 tasks load-pypi
Apr 20 10:08:15 scheduler0 swh[1499079]: INFO:swh.scheduler.celery_backend.runner:Grabbed 296 tasks index-origin-metadata
Apr 20 10:13:57 scheduler0 swh[1499079]: INFO:swh.scheduler.celery_backend.runner:Grabbed 273 tasks index-origin-metadata

Against (pre 0.13 scheduler version):

Apr 20 09:58:49 scheduler0 swh[19642]: INFO:swh.scheduler.celery_backend.runner:Grabbed 10 tasks index-origin-metadata
Apr 20 09:59:01 scheduler0 swh[19642]: INFO:swh.scheduler.celery_backend.runner:Grabbed 10 tasks index-origin-metadata
Apr 20 09:59:14 scheduler0 swh[19642]: INFO:swh.scheduler.celery_backend.runner:Grabbed 11 tasks index-origin-metadata
Apr 20 09:59:26 scheduler0 swh[19642]: INFO:swh.scheduler.celery_backend.runner:Grabbed 11 tasks index-origin-metadata
Apr 20 09:59:38 scheduler0 swh[19642]: INFO:swh.scheduler.celery_backend.runner:Grabbed 10 tasks index-origin-metadata
Apr 20 09:59:50 scheduler0 swh[19642]: INFO:swh.scheduler.celery_backend.runner:Grabbed 9 tasks index-origin-metadata
Apr 20 10:00:02 scheduler0 swh[19642]: INFO:swh.scheduler.celery_backend.runner:Grabbed 9 tasks index-origin-metadata

Well, to be fair, it seems to grab more tasks now though...

ardumont added a revision: D5561: Simplify priority computation in tests + improve exhaustivity.Apr 20 2021, 12:55 PM
vlorentz added a commit: rDSCHf33f74346f33: Simplify priority computation in tests + improve exhaustivity.Apr 20 2021, 1:01 PM
ardumont added a comment.Apr 20 2021, 3:48 PM

Somewhat good news, the scheduling db was missing an index [1]

Adding it makes the scheduler runner way more snappier (and happy).

[1]

"task_type_next_run_idx" btree (type, next_run) WHERE status = 'next_run_not_scheduled'::task_status
ardumont mentioned this in rDSCHf186910c43c5: Add default index to task(type, next_run) in schema.Apr 20 2021, 3:53 PM
ardumont added a comment.Apr 20 2021, 3:57 PM

Deployed in production as well (it was no problem for it as the index used was there already).

ardumont closed this task as Resolved.Apr 20 2021, 3:57 PM
ardumont claimed this task.
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:34 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal