Page MenuHomeSoftware Heritage

Respect task configuration to allow ignoring task result event
AbandonedPublic

Authored by ardumont on Mon, Oct 4, 5:49 PM.

Details

Summary

This is needed to actually deactivate sending their result for most of the recurring
tasks to the listener.

Following a discussion we got at some point with @olasd.

Related to T3458

Test Plan
  • tox (happy)
  • This got actually tested in staging (with task_ignore_result set to True in the loader configuration)

@vsellier hence why puppet got deactivated in staging at some point ^ (to test that)

Diff Detail

Repository
rDSCH Scheduling utilities
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 24199
Build 37778: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 37777: arc lint + arc unit

Event Timeline

ardumont edited the summary of this revision. (Show Details)
ardumont added 1 blocking reviewer(s): olasd.

Build is green

Patch application report for D6405 (id=23282)

Rebasing onto ecc14007aa...

Current branch diff-target is up to date.
Changes applied before test
commit edb5b860959911468ca35f52887ea58fe32ad16e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Sat Oct 2 18:46:51 2021 +0200

    Respect task configuration to allow ignoring task result event
    
    This is needed to actually deactivate sending their result for most of the recurring
    tasks to the listener.
    
    Related to T3458

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/452/ for more details.

This looks like an okay thing to do, but instead of only ignoring results (which would only cut down a third of the messages), we should probably be deactivating events completely on these workers.

This revision is now accepted and ready to land.Wed, Oct 6, 4:41 PM

This looks like an okay thing to do, but instead of only ignoring results (which would only cut down a third of the messages), we should probably be deactivating events completely on these workers.

Yes, I started with that config because i did not initially found the way to configure the send_events to False (or something).
After that, I found other references about configuration that could help for those, either [1] or [2].

I don't know which to choose.

[1] https://docs.celeryproject.org/en/stable/userguide/configuration.html#task-track-started

[2] https://docs.celeryproject.org/en/stable/userguide/configuration.html#std-setting-task_send_sent_event

This looks like an okay thing to do, but instead of only ignoring results (which would only cut down a third of the messages), we should probably be deactivating events completely on these workers.

Yes, I started with that config because i did not initially found the way to configure the send_events to False (or something).

It's a flag in the celery worker command line, not a config option. We set it in the systemd unit file for swh-worker@.service, and we should devise a way to turn that off for "recurrent" workers.

After that, I found other references about configuration that could help for those, either [1] or [2].

I don't know which to choose.

[1] https://docs.celeryproject.org/en/stable/userguide/configuration.html#task-track-started

[2] https://docs.celeryproject.org/en/stable/userguide/configuration.html#std-setting-task_send_sent_event

This last event is generated by the swh-scheduler-runner, not by the worker. So, with the new gen scheduler, it shouldn't be generated at all (?)

Superseded by D6439 (provided it works)