Page MenuHomeSoftware Heritage
Differential D5531

counters: Configure staging and webapp1 to use swh-counters
ClosedPublic
Actions

Authored by vsellier on Apr 15 2021, 10:23 AM.

Details

Reviewers
ardumont
Group Reviewers
System administrators
Maniphest Tasks
T3250: Deploy the counters in production (step1 webapp1 only)
Commits
rSPSITE6fbdb75919c1: counters: Configure staging and webapp1 to use swh-counters
Summary
  • Move the swh-counters's history configuration to a top level configuration as it need to be deployed on all the environments
  • Configure staging's webapp and webapp1.production to use swh-counters as the source of the counters
  • Add a history file resulting of the merge of the old history-counters-munin.json and prometheus data issued from the sql_swh_archive_object_count collection until 2021-04-15
  • Schedule the regularly refresh of the history file in production. it will be initially used by webapp1 only

Related to T3250

Test Plan
  • Moma:
*******************************************
  File[/etc/softwareheritage/web/web.yml] =>
   parameters =>
     content =>
      @@ -17,4 +17,8 @@
         cls: remote
         url: http://saatchi.internal.softwareheritage.org:5008/
      +counters:
      +  cls: remote
      +  url: http://counters1.internal.softwareheritage.org:5011/
      +counters_backend: swh-storage        <------------------------ this one is important
       log_dir: "/var/log/softwareheritage/webapp"
       secret_key: swh::deploy::webapp::conf::secret_key
*******************************************
*** End octocatalog-diff on moma.softwareheritage.org
  • webapp1.production
diff origin/production/webapp1.internal.softwareheritage.org current/webapp1.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/web/web.yml] =>
   parameters =>
     content =>
      @@ -17,4 +17,8 @@
         cls: remote
         url: http://saatchi.internal.softwareheritage.org:5008/
      +counters:
      +  cls: remote
      +  url: http://counters1.internal.softwareheritage.org:5011/
      +counters_backend: swh-counters
       log_dir: "/var/log/softwareheritage/webapp"
       secret_key: swh::deploy::webapp::conf::secret_key
      @@ -81,5 +85,5 @@
         server_url: https://auth.softwareheritage.org/auth/
         realm_name: SoftwareHeritage
      -history_counters_url: https://stats.export.softwareheritage.org/history_counters.json
      +history_counters_url: http://counters1.internal.softwareheritage.org:5011/counters_history/history.json
       allowed_hosts:
       - webapp1.internal.softwareheritage.org
*******************************************
  • webapp0.staging
diff origin/production/webapp.internal.staging.swh.network current/webapp.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/web/web.yml] =>
   parameters =>
     content =>
      @@ -19,4 +19,8 @@
         args:
           url: http://scheduler0.internal.staging.swh.network:5008/
      +counters:
      +  cls: remote
      +  url: http://counters0.internal.staging.swh.network:5011/
      +counters_backend: swh-counters
       log_dir: "/var/log/softwareheritage/webapp"
       secret_key: swh::deploy::webapp::conf::secret_key
*******************************************
  • counters0.staging
diff origin/production/counters0.internal.staging.swh.network current/counters0.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/counters/server.yml] =>
   parameters =>
     content =>
      @@ -9,5 +9,5 @@
         live_data_start: 1609462861
         cache_base_directory: "/srv/softwareheritage/counters"
      -  interval: 24h
      +  interval: 12h
         labels:
           environment: staging
*******************************************
*** End octocatalog-diff on counters0.internal.staging.swh.network
  • counters1.production
diff origin/production/counters1.internal.softwareheritage.org current/counters1.internal.softwareheritage.org
*******************************************
+ Concat_file[profile::cron::default] =>
   parameters =>
     "group": "root",
     "mode": "0644",
     "owner": "root",
     "path": "/etc/puppet-cron.d/default",
     "tag": "profile::cron::default"
*******************************************
+ Concat_fragment[profile::cron::default::_header] =>
   parameters =>
     "content": "# Managed by puppet (module profile::cron), manual changes will ...
     "order": "00",
     "tag": "profile::cron::default",
     "target": "profile::cron::default"
*******************************************
+ Concat_fragment[profile::cron::refresh_counters_cache] =>
   parameters =>
     "content": "# Cron snippet refresh_counters_cache\n0 */4 * * * swhstorage ch...
     "order": "10",
     "tag": "profile::cron::default",
     "target": "profile::cron::default"
*******************************************
+ File[/etc/cron.d/puppet-default] =>
   parameters =>
     "ensure": "link",
     "target": "/etc/puppet-cron.d/default"
*******************************************
  File[/etc/softwareheritage/counters/server.yml] =>
   parameters =>
     content =>
      @@ -3,3 +3,12 @@
         cls: redis
         host: localhost:6379
      +history:
      +  cls: prometheus
      +  prometheus_host: pergamon.internal.softwareheritage.org
      +  prometheus_port: 9090
      +  live_data_start: 1618415227
      +  cache_base_directory: "/srv/softwareheritage/counters"
      +  interval: 12h
      +  labels:
      +    environment: production
      _
*******************************************
+ File[/srv/softwareheritage/counters/static_history.json] =>
   parameters =>
     "content": "{\"content\": [[1441065600000, 0], [1444348800000, 220307136.07]...
     "ensure": "present",
     "group": "swhstorage",
     "mode": "0755",
     "owner": "swhstorage"
*******************************************
+ Profile::Cron::D[refresh_counters_cache] =>
   parameters =>
     "command": "chronic /usr/local/bin/refresh_counters_cache.sh history.json st...
     "hour": "*/4",
     "minute": 0,
     "target": "default",
     "unique_tag": "refresh_counters_cache",
     "user": "swhstorage"
*******************************************
+ Profile::Cron::File[default] =>
   parameters =>
     "target": "default"
*******************************************
*** End octocatalog-diff on counters1.internal.softwareheritage.org

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vsellier requested review of this revision.Apr 15 2021, 10:23 AM
vsellier created this revision.
Harbormaster completed remote builds in B20761: Diff 19740.Apr 15 2021, 10:23 AM
vsellier updated this revision to Diff 19744.Apr 15 2021, 10:29 AM

fix a typo on the commit description

Harbormaster completed remote builds in B20764: Diff 19744.Apr 15 2021, 10:29 AM
vsellier edited the summary of this revision. (Show Details)Apr 15 2021, 10:30 AM
vsellier edited the test plan for this revision. (Show Details)Apr 15 2021, 10:33 AM
ardumont accepted this revision.Apr 15 2021, 10:37 AM
ardumont added a subscriber: ardumont.

lgtm

one remark inline ;)

data/common/common.yaml
3283

This may be decreased to be more frequent.
AFAIU the counter computations will be way faster than the pg counters ;)

This revision is now accepted and ready to land.Apr 15 2021, 10:37 AM
vsellier added inline comments.Apr 15 2021, 11:06 AM
data/common/common.yaml
3283

This value is the interval used to request the values from prometheus.
The refresh by itself is done via a cron, scheduled each 4h

Closed by commit rSPSITE6fbdb75919c1: counters: Configure staging and webapp1 to use swh-counters (authored by vsellier). · Explain WhyApr 15 2021, 11:11 AM
This revision was automatically updated to reflect the committed changes.
vsellier added a commit: rSPSITE6fbdb75919c1: counters: Configure staging and webapp1 to use swh-counters.

Revision Contents
Changeset List

PathSize
data/
common/
24 lines
deployments/
staging/
19 lines
hostname/
3 lines
site-modules/
profile/
files/
swh/
deploy/
counters/
static_history/
3 lines
manifests/
swh/
deploy/
13 lines

Diff 19748

View Options

data/common/common.yaml

Loading...
View Options

data/deployments/staging/common.yaml

Loading...
View Options

data/hostname/webapp1.internal.softwareheritage.org.yaml

Loading...
View Options

site-modules/profile/files/swh/deploy/counters/static_history/static_history.json

Loading...
View Options

site-modules/profile/manifests/swh/deploy/counters.pp

Loading...
About · Developer information · Legal