Differential D5470

staging: configure counters history pipeline
ClosedPublic
Actions

Authored by vsellier on Apr 9 2021, 9:47 AM.

Details

Reviewers

ardumont

Group Reviewers

System administrators

Maniphest Tasks

T3215: Deploy the new counters in staging

Commits

rSPSITE52709f7a0330: staging: configure counters history pipeline

Summary

Add the history section on the backend configurartion
add a cron to regularly refresh the history cache

Related to T3215

Test Plan

Moma: declare the default value for the history File

diff origin/production/moma.softwareheritage.org current/moma.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/web/web.yml] =>
   parameters =>
     content =>
      @@ -81,4 +81,5 @@
         server_url: https://auth.softwareheritage.org/auth/
         realm_name: SoftwareHeritage
      +history_counters_url: https://stats.export.softwareheritage.org/history_counters.json
       allowed_hosts:
       - archive.softwareheritage.org
*******************************************

Staging: Declare the history section in the configuration and add the crontab crontab

diff origin/production/counters0.internal.staging.swh.network current/counters0.internal.staging.swh.network
*******************************************
+ Concat_file[profile::cron::default] =>
   parameters =>
     "group": "root",
     "mode": "0644",
     "owner": "root",
     "path": "/etc/puppet-cron.d/default",
     "tag": "profile::cron::default"
*******************************************
+ Concat_fragment[profile::cron::default::_header] =>
   parameters =>
     "content": "# Managed by puppet (module profile::cron), manual changes will ...
     "order": "00",
     "tag": "profile::cron::default",
     "target": "profile::cron::default"
*******************************************
+ Concat_fragment[profile::cron::refresh_counters_cache] =>
   parameters =>
     "content": "# Cron snippet refresh_counters_cache\n0 */4 * * * swhstorage ch...
     "order": "10",
     "tag": "profile::cron::default",
     "target": "profile::cron::default"
*******************************************
+ File[/etc/cron.d/puppet-default] =>
   parameters =>
     "ensure": "link",
     "target": "/etc/puppet-cron.d/default"
*******************************************
  File[/etc/softwareheritage/counters/server.yml] =>
   parameters =>
     content =>
      @@ -3,3 +3,10 @@
         cls: redis
         host: localhost:6379
      +history:
      +  cls: prometheus
      +  prometheus_host: pergamon.internal.softwareheritage.org
      +  prometheus_port: 9090
      +  live_data_start: 1609462861
      +  cache_base_directory: "/srv/softwareheritage/counters"
      +  interval: 24h
      _
*******************************************
+ File[/srv/softwareheritage/counters] =>
   parameters =>
     "ensure": "directory",
     "group": "swhstorage",
     "mode": "0775",
     "owner": "swhstorage"
*******************************************
+ File[/usr/local/bin/refresh_counters_cache.sh] =>
   parameters =>
     "content": "#!/bin/bash\n\n##\n# File managed by puppet (class profile::swh:...
     "ensure": "present",
     "group": "swhstorage",
     "mode": "0755",
     "owner": "swhstorage"
*******************************************
+ Profile::Cron::D[refresh_counters_cache] =>
   parameters =>
     "command": "chronic /usr/local/bin/refresh_counters_cache.sh history.json ",...
     "hour": "*/4",
     "minute": 0,
     "target": "default",
     "unique_tag": "refresh_counters_cache",
     "user": "swhstorage"
*******************************************
+ Profile::Cron::File[default] =>
   parameters =>
     "target": "default"
*******************************************
*** End octocatalog-diff on counters0.internal.staging.swh.network

Production: Just install the script used by the crontab but don't schedule it

diff origin/production/counters1.internal.softwareheritage.org current/counters1.internal.softwareheritage.org
*******************************************
+ File[/srv/softwareheritage/counters] =>
   parameters =>
     "ensure": "directory",
     "group": "swhstorage",
     "mode": "0775",
     "owner": "swhstorage"
*******************************************
+ File[/usr/local/bin/refresh_counters_cache.sh] =>
   parameters =>
     "content": "#!/bin/bash\n\n##\n# File managed by puppet (class profile::swh:...
     "ensure": "present",
     "group": "swhstorage",
     "mode": "0755",
     "owner": "swhstorage"
*******************************************
*** End octocatalog-diff on counters1.internal.softwareheritage.org