Page MenuHomeSoftware Heritage

scheduler: Update configuration according to latest development
ClosedPublic

Authored by ardumont on Feb 12 2019, 1:58 PM.

Details

Summary
  • listener: Migrate to use the swh-scheduler cli as entry point
  • runner: Migrate to use the swh-scheduler cli as entry point
  • scheduler-backend: Fix configuration key change
  • scheduler: Factorize configuration for listener/runner/api
  • Add missing celery configuration for listener/runner

Related T1525

Test Plan
$ cd puppet-environment
$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to update_scheduler_config saatchi
Found host saatchi.internal.softwareheritage.org
Cloning into '/tmp/swh-ocd.LqmOZ9ED/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.LqmOZ9ED/environments/update_scheduler_config/data/private'...
done.
*** Running octocatalog-diff on host saatchi.internal.softwareheritage.org
I, [2019-02-13T16:21:49.260256 #11216]  INFO -- : Catalogs compiled for saatchi.internal.softwareheritage.org
I, [2019-02-13T16:21:50.026120 #11216]  INFO -- : Diffs computed for saatchi.internal.softwareheritage.org
diff origin/production/saatchi.internal.softwareheritage.org current/saatchi.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/backend/scheduler.yml] =>
   parameters =>
     content =>
      @@ -3,4 +3,4 @@
         cls: local
         args:
      -    scheduling_db: host=db dbname=softwareheritage-scheduler user=swhscheduler password=swh::deploy::scheduler::db::password
      +    db: host=db dbname=softwareheritage-scheduler user=swhscheduler password=swh::deploy::scheduler::db::password
      _
*******************************************
- File[/etc/softwareheritage/scheduler.ini]
*******************************************
+ File[/etc/softwareheritage/scheduler.yml] =>
   parameters =>
      "ensure": "present"
      "group": "swhscheduler"
      "mode": "0640"
      "notify": ["Service[swh-scheduler-listener]", "Service[swh-scheduler-runner]"]
      "owner": "root"
      "content": >>>
---
scheduler:
  cls: local
  args:
    db: host=db dbname=softwareheritage-scheduler user=swhscheduler password=swh::deploy::scheduler::db::password
celery:
  task_broker: amqp://swhproducer:swh::deploy::scheduler::task_broker::password@rabbitmq:5672//
<<<
*******************************************
  File[/etc/systemd/system/swh-scheduler-listener.service] =>
   parameters =>
     content =>
      @@ -12,5 +12,5 @@
       Group=swhscheduler
       Type=simple
      -ExecStart=/usr/bin/python3 -m swh.scheduler.celery_backend.listener
      +ExecStart=/usr/bin/swh-scheduler --config-file /etc/softwareheritage/scheduler.yml --log-level INFO listener
       Restart=always
       RestartSec=10
*******************************************
  File[/etc/systemd/system/swh-scheduler-runner.service] =>
   parameters =>
     content =>
      @@ -12,5 +12,5 @@
       Group=swhscheduler
       Type=simple
      -ExecStart=/bin/sh -c 'while true; do echo running pending tasks at `/bin/date`...; /usr/bin/python3 -m swh.scheduler.celery_backend.runner; sleep 10; done'
      +ExecStart=/usr/bin/swh-scheduler --config-file /etc/softwareheritage/scheduler.yml --log-level INFO runner --period 10
       Restart=always
       RestartSec=10
*******************************************
  Systemd::Unit_file[swh-scheduler-listener.service] =>
   parameters =>
     content =>
      @@ -12,5 +12,5 @@
       Group=swhscheduler
       Type=simple
      -ExecStart=/usr/bin/python3 -m swh.scheduler.celery_backend.listener
      +ExecStart=/usr/bin/swh-scheduler --config-file /etc/softwareheritage/scheduler.yml --log-level INFO listener
       Restart=always
       RestartSec=10
*******************************************
  Systemd::Unit_file[swh-scheduler-runner.service] =>
   parameters =>
     content =>
      @@ -12,5 +12,5 @@
       Group=swhscheduler
       Type=simple
      -ExecStart=/bin/sh -c 'while true; do echo running pending tasks at `/bin/date`...; /usr/bin/python3 -m swh.scheduler.celery_backend.runner; sleep 10; done'
      +ExecStart=/usr/bin/swh-scheduler --config-file /etc/softwareheritage/scheduler.yml --log-level INFO runner --period 10
       Restart=always
       RestartSec=10
*******************************************
*** End octocatalog-diff on saatchi.internal.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont added inline comments.
site-modules/profile/manifests/swh/deploy/scheduler.pp
120

for now, i did not change this as i'm not sure yet how to use /usr/bin/swh-scheduler api-server and gunicorn together.

  • defaults: Add missing configuration for listener/runner
ardumont edited the summary of this revision. (Show Details)
ardumont added inline comments.
data/defaults.yaml
1725

That's really the configuration for the listener/runner so that might be best to rename this appropriately to simplify maintenance.

data/hostname/saatchi.internal.softwareheritage.org.yaml
6 ↗(On Diff #3544)

by default, all remote configuration should discuss with the scheduler's api (that's the default value for swh::deploy::scheduler::remote::config).
Except for the local instance running on saatchi.

  • scheduler: Factorize configuration for listener/runner/api
ardumont marked an inline comment as done.
ardumont added inline comments.
site-modules/profile/manifests/swh/deploy/scheduler.pp
120

As discussed orally, this does not need to change.
What needs to change is the implicit configuration which should go away.
This could be replaced for example by a variable environment targeting the configuration file used by the server.
This is not a requisite for the diff.

LGTM, but hey, these are puppets!

This revision is now accepted and ready to land.Feb 14 2019, 9:39 AM
This revision was automatically updated to reflect the committed changes.