Page MenuHomeSoftware Heritage

Stop sending next-gen scheduled task results to scheduler listener
AbandonedPublic

Authored by ardumont on Oct 5 2021, 9:18 AM.

Details

Summary

This is 3-fold, this should:

  • ease the listener's work to only deal with required tasks (the ones that are still

scheduled through the initial scheduler-runner, e.g. save code now, deposit, vault,
...). The other tasks scheduled with the next-gen scheduler no longer needs to report
such status as a dedicated journal client is in charge of that part.

  • decrease the load induced by those update in the scheduler db
  • help the save code now have better status report (it's still using the scheduler db to

access some of those status as fallback).

This is the other side of the configuration adaptation done in the swh.scheduler.task
module [1]

Related to T3458
[1] Related to D6405

Test Plan

Run staging scheduler0 with some loader npm patched with this configuration.

The "task-result" even is still received but no longer processed, so no more writes
happen to the scheduler backend (the part updating the result in db, the scheduler
journal client is in charge of that).

bin/octocatalog-diff

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker01
I, [2021-10-05T09:16:49.356428 #909179]  INFO -- : Catalogs compiled for worker01.softwareheritage.org
I, [2021-10-05T09:16:49.691323 #909179]  INFO -- : Diffs computed for worker01.softwareheritage.org
diff origin/production/worker01.softwareheritage.org current/worker01.softwareheritage.org
  File[/etc/softwareheritage/loader_archive.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.archive.tasks.LoadArchive
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_cran.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.cran.tasks.LoadCRAN
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_debian.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.debian.tasks.LoadDebian
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_deposit.yml] =>
   parameters =>
     content =>
      @@ -20,4 +20,5 @@
         task_queues:
         - swh.loader.package.deposit.tasks.LoadDeposit
      +  task_ignore_result: false
       deposit:
         url: https://deposit.softwareheritage.org/1/private/
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -23,3 +23,4 @@
         - swh.loader.git.tasks.LoadDiskGitRepository
         - swh.loader.git.tasks.UncompressAndLoadDiskGitRepository
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_high_priority.yml] =>
   parameters =>
     content =>
      @@ -26,3 +26,4 @@
         - save_code_now:swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
         - save_code_now:swh.loader.package.archive.tasks.LoadArchive
      +  task_ignore_result: false
      _
*******************************************
  File[/etc/softwareheritage/loader_mercurial.yml] =>
   parameters =>
     content =>
      @@ -22,3 +22,4 @@
         - swh.loader.mercurial.tasks.LoadMercurial
         - swh.loader.mercurial.tasks.LoadArchiveMercurial
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_nixguix.yml] =>
   parameters =>
     content =>
      @@ -20,4 +20,5 @@
         task_queues:
         - swh.loader.package.nixguix.tasks.LoadNixguix
      +  task_ignore_result: false
       unsupported_file_extensions:
       - patch
*******************************************
  File[/etc/softwareheritage/loader_npm.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.npm.tasks.LoadNpm
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_opam.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.opam.tasks.LoadOpam
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_pypi.yml] =>
   parameters =>
     content =>
      @@ -20,3 +20,4 @@
         task_queues:
         - swh.loader.package.pypi.tasks.LoadPyPI
      +  task_ignore_result: true
      _
*******************************************
  File[/etc/softwareheritage/loader_svn.yml] =>
   parameters =>
     content =>
      @@ -22,3 +22,4 @@
         - swh.loader.svn.tasks.MountAndLoadSvnRepository
         - swh.loader.svn.tasks.DumpMountAndLoadSvnRepository
      +  task_ignore_result: true
      _
*******************************************
*** End octocatalog-diff on worker01.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 24202
Build 37781: arc lint + arc unit

Event Timeline

ardumont created this revision.

Rather than doing this, we should probably disable worker task events altogether (that is, run celery worker without the --events/--task-events flag)

In D6408#166674, @olasd wrote:

Rather than doing this, we should probably disable worker task events altogether (that is, run celery worker without the --events/--task-events flag)

Your other comment [1] and this one makes total sense.
Thanks.

Not that it is simple to implement (i have a diff locally and then octodiff made me realize that the swh-worker@<service>.service is actually linked to swh-worker@.service...
So modifying the template is not enough since that flip flops between the services (the one needing it and the others...)

Hence the "devise a way" sentence you used, probably ;)

[1] D6405#166755

Superseded by D6439 (provided it works)