Page MenuHomeSoftware Heritage

Install update-metrics as a service called daily
ClosedPublic

Authored by ardumont on Aug 3 2021, 2:25 PM.

Details

Summary

As an implementation detail, the service is called daily through a systemd timer.

This updates the scheduler_metrics table regularly so the distribution per forge stays
up-to-date. This also avoid having this running in a tmux session on saatchi. This also
deals with properly update such metrics on the staging infra.

[1] P1113

Related to T3127

Test Plan

tests within vagrant as well:

root@scheduler0:~# systemctl status swh-scheduler-update-metrics.timer
● swh-scheduler-update-metrics.timer - regular scheduler update-metrics refresh
   Loaded: loaded (/etc/systemd/system/swh-scheduler-update-metrics.timer; enabled; vendor preset: enabled)
   Active: active (waiting) since Tue 2021-08-03 14:29:18 UTC; 1min 6s ago
  Trigger: Wed 2021-08-04 00:00:00 UTC; 9h left

Aug 03 14:29:18 scheduler0 systemd[1]: Started regular scheduler update-metrics refresh.
root@scheduler0:~# systemctl status swh-scheduler-update-metrics.service
● swh-scheduler-update-metrics.service - Software Heritage scheduler update-metrics
   Loaded: loaded (/etc/systemd/system/swh-scheduler-update-metrics.service; disabled; vendor preset: enabled)
   Active: inactive (dead)

octo diff on scheduler nodes, this wants to install the timer and service (only
scheduler0.staging, same goes for saatchi):

$ bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging scheduler0.internal.staging.swh.network
Found host scheduler0.internal.staging.swh.network
WARN     -> Environment "wip-pg-hba-rules-in-yaml" contained non-word characters, correcting name to wip_pg_hba_rules_in_yaml
Cloning into '/tmp/swh-ocd.CKGGPcLa/environments/production/data/private'...
done.
Cloning into '/tmp/swh-ocd.CKGGPcLa/environments/staging/data/private'...
done.
*** Running octocatalog-diff on host scheduler0.internal.staging.swh.network
I, [2021-08-03T16:46:44.068868 #12855]  INFO -- : Catalogs compiled for scheduler0.internal.staging.swh.network
I, [2021-08-03T16:46:46.554274 #12855]  INFO -- : Diffs computed for scheduler0.internal.staging.swh.network
diff origin/production/scheduler0.internal.staging.swh.network current/scheduler0.internal.staging.swh.network
*******************************************
  File[/etc/systemd/system/prometheus-node-exporter.service.d/restart.conf] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
  File[/etc/systemd/system/prometheus-node-exporter.service.d] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
  File[/etc/systemd/system/prometheus-statsd-exporter.service.d/restart.conf] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
  File[/etc/systemd/system/prometheus-statsd-exporter.service.d] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
  File[/etc/systemd/system/rabbitmq-server.service.d/90-limits.conf] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
  File[/etc/systemd/system/rabbitmq-server.service.d] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
+ File[/etc/systemd/system/swh-scheduler-update-metrics.service] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": "Class[Systemd::Systemctl::Daemon_reload]"
      "owner": "root"
      "show_diff": true
      "content": >>>
# Scheduler Update Metrics unit file
# Managed by puppet class profile::swh::deploy::scheduler
# Changes will be overwritten

[Unit]
Description=Software Heritage scheduler update-metrics
Requires=network.target
After=network.target

[Service]
User=swhscheduler
Group=swhscheduler
Type=simple
ExecStart=/usr/bin/swh scheduler --config-file /etc/softwareheritage/scheduler/listener-runner.yml origin update-metrics

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ File[/etc/systemd/system/swh-scheduler-update-metrics.timer] =>
   parameters =>
      "ensure": "file"
      "group": "root"
      "mode": "0444"
      "notify": ["Class[Systemd::Systemctl::Daemon_reload]", "Service[swh-scheduler-update-metrics.timer]"]
      "owner": "root"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Unit=swh-scheduler-update-metrics.service

[Unit]
Description=regular scheduler update-metrics refresh
<<<
*******************************************
+ Service[swh-scheduler-update-metrics.timer] =>
   parameters =>
      "enable": true
      "ensure": true
      "provider": "systemd"
*******************************************
  Systemd::Dropin_file[prometheus-node-exporter/restart.conf] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
  Systemd::Dropin_file[prometheus-statsd-exporter/restart.conf] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
  Systemd::Dropin_file[rabbitmq-server.service-90-limits.conf] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
  Systemd::Service_limits[rabbitmq-server.service] =>
   parameters =>
     selinux_ignore_defaults =>
      + false
*******************************************
+ Systemd::Timer[swh-scheduler-update-metrics.timer] =>
   parameters =>
      "active": true
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "service_content": >>>
# Scheduler Update Metrics unit file
# Managed by puppet class profile::swh::deploy::scheduler
# Changes will be overwritten

[Unit]
Description=Software Heritage scheduler update-metrics
Requires=network.target
After=network.target

[Service]
User=swhscheduler
Group=swhscheduler
Type=simple
ExecStart=/usr/bin/swh scheduler --config-file /etc/softwareheritage/scheduler/listener-runner.yml origin update-metrics

[Install]
WantedBy=multi-user.target
<<<
      "timer_content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Unit=swh-scheduler-update-metrics.service

[Unit]
Description=regular scheduler update-metrics refresh
<<<
*******************************************
  Systemd::Tmpfile[gunicorn-swh-scheduler.conf] =>
   parameters =>
     filename =>
      + gunicorn-swh-scheduler.conf
*******************************************
  Systemd::Tmpfile[gunicorn.conf] =>
   parameters =>
     filename =>
      + gunicorn.conf
*******************************************
+ Systemd::Unit_file[swh-scheduler-update-metrics.service] =>
   parameters =>
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
# Scheduler Update Metrics unit file
# Managed by puppet class profile::swh::deploy::scheduler
# Changes will be overwritten

[Unit]
Description=Software Heritage scheduler update-metrics
Requires=network.target
After=network.target

[Service]
User=swhscheduler
Group=swhscheduler
Type=simple
ExecStart=/usr/bin/swh scheduler --config-file /etc/softwareheritage/scheduler/listener-runner.yml origin update-metrics

[Install]
WantedBy=multi-user.target
<<<
*******************************************
+ Systemd::Unit_file[swh-scheduler-update-metrics.timer] =>
   parameters =>
      "active": true
      "enable": true
      "ensure": "present"
      "group": "root"
      "mode": "0444"
      "owner": "root"
      "path": "/etc/systemd/system"
      "show_diff": true
      "content": >>>
[Install]
WantedBy=timers.target

[Timer]
OnCalendar=daily
Unit=swh-scheduler-update-metrics.service

[Unit]
Description=regular scheduler update-metrics refresh
<<<
*******************************************
*** End octocatalog-diff on scheduler0.internal.staging.swh.network

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 22867
Build 35663: arc lint + arc unit

Event Timeline

ardumont created this revision.

If there any log in output? Is-it possible to monitor the duration of the command and most importantly avoiding it to run several times in parallel?

I have just saw the paste so I have the response to my first question ;)

If there any log in output? Is-it possible to monitor the duration of the command

There is actually an output which is filtered.

Something like:

...
    {
        "last_update": "2021-08-03 12:23:11.230366+00:00",
        "lister_id": "ff34a2b5-2e81-4566-9627-61fab06f8f52",
        "origins_enabled": "1036",
        "origins_known": "1036",
        "origins_never_visited": "1025",
        "origins_with_pending_changes": "0",
        "visit_type": "git"
    },
    {
        "last_update": "2021-08-03 12:23:11.230366+00:00",
        "lister_id": "59354ffc-0a34-4140-8503-5f398a763097",
        "origins_enabled": "1091",
        "origins_known": "1091",
        "origins_never_visited": "282",
        "origins_with_pending_changes": "11",
        "visit_type": "git"
    },
    {
        "last_update": "2021-08-03 12:23:11.230366+00:00",
        "lister_id": "31df8830-df11-4120-8e76-323981a9c88d",
        "origins_enabled": "189",
        "origins_known": "189",
        "origins_never_visited": "16",
        "origins_with_pending_changes": "0",
        "visit_type": "git"
    }
]

and most importantly avoiding it to run several times in parallel?

as most likely like what suggests T3451.

Ah interesting.
Let's try and do that ;)

https://www.freedesktop.org/software/systemd/man/systemd.timer.html#

It looks like it solves the concurrency issue and will allow to keep the logs

Implement the diff as systemd service/timer instead of a cron. This also installs said
service for a daily execution instead of hourly (as per the related task discussion).

What i just said from the appropriate repository

Puppetfile
188

The systemd timer is only supported from 2.8 onward.

Use the most recent version still supporting puppet 5 (from 3.0 tag onward, the
changelog claim it stops support from puppet 5 which we are still using).

ardumont edited the test plan for this revision. (Show Details)

Update to 2.12 (updating the fork source we host on the forge)

Drop no longer needed cron part.

ardumont retitled this revision from Add update-metrics cli call cron on scheduler nodes to Install update-metrics as a service called daily.Aug 3 2021, 4:45 PM
ardumont edited the summary of this revision. (Show Details)

LGTM

It seems the systemd module upgrade add an internal change :

*******************************************
  Systemd::Service_limits[rabbitmq-server.service] =>
   parameters =>
     selinux_ignore_defaults =>
      + false

but as we have checked together, it seems it has no impacts on the service configuration / content of the dropin files

This revision is now accepted and ready to land.Aug 3 2021, 4:46 PM