Differential D6873

worker-large: Make worker consume standard git queue
ClosedPublic
Actions

Authored by ardumont on Jan 3 2022, 12:39 PM.

Details

Reviewers

olasd

Group Reviewers

System administrators

Maniphest Tasks

T3825: Update large workers to consume standard git queue

Commits

rSPSITEdd9193636941: worker-large: Make worker consume standard git queue

Summary

Related to T3825

Test Plan

currently applied in workers to check the load is ok, it is [1]

[1]

octo-diff:

$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker18
Found host worker18.softwareheritage.org
...
I, [2022-01-03T12:37:48.377541 #177099]  INFO -- : Catalogs compiled for worker18.softwareheritage.org
I, [2022-01-03T12:37:48.659271 #177099]  INFO -- : Diffs computed for worker18.softwareheritage.org
diff origin/production/worker18.softwareheritage.org current/worker18.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -26,3 +26,4 @@
         task_queues:
         - oneshot3:swh.loader.git.tasks.UpdateGitRepository
      +  - swh.loader.git.tasks.UpdateGitRepository
      _
*******************************************
  File[/etc/systemd/system/swh-worker@loader_git.service.d/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=7
      +Environment=CONCURRENCY=10
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  File[/etc/systemd/system/swh-worker@loader_oneshot.service.d/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=7
      +Environment=CONCURRENCY=5
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  Systemd::Dropin_file[swh-worker@loader_git/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=7
      +Environment=CONCURRENCY=10
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  Systemd::Dropin_file[swh-worker@loader_oneshot/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=7
      +Environment=CONCURRENCY=5
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
*** End octocatalog-diff on worker18.softwareheritage.org

$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging worker17
...
I, [2022-01-03T12:37:59.397696 #178766]  INFO -- : Catalogs compiled for worker17.softwareheritage.org
I, [2022-01-03T12:37:59.678886 #178766]  INFO -- : Diffs computed for worker17.softwareheritage.org
diff origin/production/worker17.softwareheritage.org current/worker17.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/loader_git.yml] =>
   parameters =>
     content =>
      @@ -26,3 +26,4 @@
         task_queues:
         - oneshot3:swh.loader.git.tasks.UpdateGitRepository
      +  - swh.loader.git.tasks.UpdateGitRepository
      _
*******************************************
  File[/etc/systemd/system/swh-worker@loader_git.service.d/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=10
      +Environment=CONCURRENCY=20
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  File[/etc/systemd/system/swh-worker@loader_oneshot.service.d/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=20
      +Environment=CONCURRENCY=10
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  Systemd::Dropin_file[swh-worker@loader_git/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=10
      +Environment=CONCURRENCY=20
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
  Systemd::Dropin_file[swh-worker@loader_oneshot/parameters.conf] =>
   parameters =>
     content =>
      @@ -3,5 +3,5 @@
      _
       [Service]
      -Environment=CONCURRENCY=20
      +Environment=CONCURRENCY=10
       Environment=MAX_TASKS_PER_CHILD=1
       Environment=LOGLEVEL=info
*******************************************
*** End octocatalog-diff on worker17.softwareheritage.org