Page MenuHomeSoftware Heritage
Differential D5854

Install the scheduler-runner-priority in scheduler nodes
ClosedPublic
Actions

Authored by ardumont on Jun 11 2021, 11:21 AM.

Details

Reviewers
olasd
Group Reviewers
System administrators
Maniphest Tasks
T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks
Commits
rSPSITE232088a7b74a: scheduler: Drop clean up instructions
rSPSITEb9de3f4ede43: Install the scheduler-runner-priority in scheduler nodes
Summary

This got already deployed in production manually.

Related to T3367

Test Plan
$ bin/octocatalog-diff --to staging saatchi 2>&1
*******************************************
  File[/etc/softwareheritage/scheduler/listener-runner.yml] =>
   parameters =>
     notify =>
      - ["Service[swh-scheduler-listener]", "Service[swh-scheduler-runner]"]
      + ["Service[swh-scheduler-listener]", "Service[swh-scheduler-runner-priority]", "Service[swh-scheduler-runner]"]
*******************************************
- File[/etc/systemd/system/swh-scheduler-runner.service]
*******************************************
+ File[/etc/systemd/system/swh-scheduler-swh-scheduler-runner-priority.service] =>
   parameters =>
     "content": "# Scheduler Runner unit file\n# Managed by puppet class profile:...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-scheduler-swh-scheduler-runner.service] =>
   parameters =>
     "content": "# Scheduler Runner unit file\n# Managed by puppet class profile:...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ Profile::Swh::Deploy::Scheduler::Runner[swh-scheduler-runner-priority] =>
   parameters =>
     "config_file": "/etc/softwareheritage/scheduler/listener-runner.yml",
     "group": "swhscheduler",
     "packages": [
       "python3-swh.scheduler"
     ],
     "priority": true,
     "sentry_dsn": "https://swh::deploy::scheduler::sentry_token@sentry.softwareh...
     "sentry_environment": "production",
     "sentry_swh_package": "swh.scheduler",
     "service_name": "swh-scheduler-runner-priority",
     "user": "swhscheduler"
*******************************************
+ Profile::Swh::Deploy::Scheduler::Runner[swh-scheduler-runner] =>
   parameters =>
     "config_file": "/etc/softwareheritage/scheduler/listener-runner.yml",
     "group": "swhscheduler",
     "packages": [
       "python3-swh.scheduler"
     ],
     "priority": false,
     "sentry_dsn": "https://swh::deploy::scheduler::sentry_token@sentry.softwareh...
     "sentry_environment": "production",
     "sentry_swh_package": "swh.scheduler",
     "service_name": "swh-scheduler-runner",
     "user": "swhscheduler"
*******************************************
+ Service[swh-scheduler-runner-priority] =>
   parameters =>
     "enable": true,
     "ensure": "running",
     "subscribe": [
       "Package[python3-swh.scheduler]"
     ]
*******************************************
- Systemd::Unit_file[swh-scheduler-runner.service]
*******************************************
+ Systemd::Unit_file[swh-scheduler-swh-scheduler-runner-priority.service] =>
   parameters =>
     "content": "# Scheduler Runner unit file\n# Managed by puppet class profile:...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "notify": "Service[swh-scheduler-runner-priority]",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
+ Systemd::Unit_file[swh-scheduler-swh-scheduler-runner.service] =>
   parameters =>
     "content": "# Scheduler Runner unit file\n# Managed by puppet class profile:...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "notify": "Service[swh-scheduler-runner]",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
*** End octocatalog-diff on saatchi.internal.softwareheritage.org
$ bin/octocatalog-diff --to staging scheduler0.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/scheduler/listener-runner.yml] =>
   parameters =>
     notify =>
      - ["Service[swh-scheduler-listener]", "Service[swh-scheduler-runner]"]
      + ["Service[swh-scheduler-listener]", "Service[swh-scheduler-runner-priority]", "Service[swh-scheduler-runner]"]
*******************************************
- File[/etc/systemd/system/swh-scheduler-runner.service]
*******************************************
+ File[/etc/systemd/system/swh-scheduler-swh-scheduler-runner-priority.service] =>
   parameters =>
     "content": "# Scheduler Runner unit file\n# Managed by puppet class profile:...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ File[/etc/systemd/system/swh-scheduler-swh-scheduler-runner.service] =>
   parameters =>
     "content": "# Scheduler Runner unit file\n# Managed by puppet class profile:...
     "ensure": "file",
     "group": "root",
     "mode": "0444",
     "notify": "Class[Systemd::Systemctl::Daemon_reload]",
     "owner": "root",
     "show_diff": true
*******************************************
+ Profile::Swh::Deploy::Scheduler::Runner[swh-scheduler-runner-priority] =>
   parameters =>
     "config_file": "/etc/softwareheritage/scheduler/listener-runner.yml",
     "group": "swhscheduler",
     "packages": [
       "python3-swh.scheduler"
     ],
     "priority": true,
     "sentry_dsn": "https://swh::deploy::scheduler::sentry_token@sentry.softwareh...
     "sentry_environment": "staging",
     "sentry_swh_package": "swh.scheduler",
     "service_name": "swh-scheduler-runner-priority",
     "user": "swhscheduler"
*******************************************
+ Profile::Swh::Deploy::Scheduler::Runner[swh-scheduler-runner] =>
   parameters =>
     "config_file": "/etc/softwareheritage/scheduler/listener-runner.yml",
     "group": "swhscheduler",
     "packages": [
       "python3-swh.scheduler"
     ],
     "priority": false,
     "sentry_dsn": "https://swh::deploy::scheduler::sentry_token@sentry.softwareh...
     "sentry_environment": "staging",
     "sentry_swh_package": "swh.scheduler",
     "service_name": "swh-scheduler-runner",
     "user": "swhscheduler"
*******************************************
+ Service[swh-scheduler-runner-priority] =>
   parameters =>
     "enable": true,
     "ensure": "running",
     "subscribe": [
       "Package[python3-swh.scheduler]"
     ]
*******************************************
- Systemd::Unit_file[swh-scheduler-runner.service]
*******************************************
+ Systemd::Unit_file[swh-scheduler-swh-scheduler-runner-priority.service] =>
   parameters =>
     "content": "# Scheduler Runner unit file\n# Managed by puppet class profile:...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "notify": "Service[swh-scheduler-runner-priority]",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
+ Systemd::Unit_file[swh-scheduler-swh-scheduler-runner.service] =>
   parameters =>
     "content": "# Scheduler Runner unit file\n# Managed by puppet class profile:...
     "ensure": "present",
     "group": "root",
     "mode": "0444",
     "notify": "Service[swh-scheduler-runner]",
     "owner": "root",
     "path": "/etc/systemd/system",
     "show_diff": true
*******************************************
*** End octocatalog-diff on scheduler0.internal.staging.swh.network

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont requested review of this revision.Jun 11 2021, 11:21 AM
ardumont created this revision.
Harbormaster completed remote builds in B21925: Diff 20944.Jun 11 2021, 11:21 AM
ardumont updated this revision to Diff 20945.Jun 11 2021, 11:22 AM

Drop notice instructions

Harbormaster completed remote builds in B21926: Diff 20945.Jun 11 2021, 11:22 AM
ardumont added inline comments.Jun 11 2021, 11:25 AM
site-modules/profile/manifests/swh/deploy/scheduler/runner.pp
42

See below to determine how the command is computed in the end (per runner).

runner (as the current one):

/usr/bin/swh --log-level INFO scheduler --config-file /etc/softwareheritage/scheduler/listener-runner.yml start-runner --period 10

runner-priority:

/usr/bin/swh --log-level INFO scheduler --config-file /etc/softwareheritage/scheduler/listener-runner.yml start-runner --period 10 --with-priority --task-type load-git --task-type load-svn --task-type load-archive-files --task-type load-hg
olasd accepted this revision.Jun 11 2021, 11:26 AM
olasd added a subscriber: olasd.
olasd added inline comments.
site-modules/profile/manifests/swh/deploy/scheduler.pp
43–51

This could go away

site-modules/profile/manifests/swh/deploy/scheduler/runner.pp
64

Not sure that's useful, as puppet doesn't upgrade the packages (afaik)

This revision is now accepted and ready to land.Jun 11 2021, 11:26 AM
ardumont updated this revision to Diff 20946.Jun 11 2021, 11:37 AM

Drop unneeded instructions

ardumont updated this revision to Diff 20947.Jun 11 2021, 11:38 AM
  • scheduler: Drop unneeded up instructions
Harbormaster completed remote builds in B21928: Diff 20947.Jun 11 2021, 11:38 AM
ardumont added inline comments.Jun 11 2021, 11:38 AM
site-modules/profile/manifests/swh/deploy/scheduler.pp
43–51

right!

swh-public-ci added a comment.Jun 11 2021, 11:40 AM

Build is green

Patch application report for D5854 (id=20946)

Rebasing onto c7707b5c83...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-364-D5854.
Changes applied before test



See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/364/ for more details.
Harbormaster completed remote builds in B21927: Diff 20946.Jun 11 2021, 11:40 AM
Closed by commit rSPSITEb9de3f4ede43: Install the scheduler-runner-priority in scheduler nodes (authored by ardumont).  ·  Explain WhyJun 11 2021, 11:41 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rSPSITEb9de3f4ede43: Install the scheduler-runner-priority in scheduler nodes.
ardumont added a commit: rSPSITE232088a7b74a: scheduler: Drop clean up instructions.
Revision Contents
Changeset List
PathSize
data/
common/
7 lines
site-modules/
profile/
manifests/
swh/
deploy/
61 lines
scheduler/
67 lines
templates/
swh/
deploy/
scheduler/
2 lines
Diff 20948
View Options
data/common/common.yaml
Loading...
View Options
site-modules/profile/manifests/swh/deploy/scheduler.pp
Loading...
View Options
site-modules/profile/manifests/swh/deploy/scheduler/runner.pp
Loading...
View Options
site-modules/profile/templates/swh/deploy/scheduler/swh-scheduler-runner.service.erb
Loading...
About · Developer information · Legal