Page MenuHomeSoftware Heritage

Install webapp counters in the staging webapp/storage
Closed, MigratedEdits Locked

Description

Just to reference that needs to happen some day.

It's only installed in the main one.

Function required for this are packaged within the swh.storage module (and already installed in the staging db).
What's missing is actual crons which trigger those count.

The diagram [1] should definitely help for this install.

[1] T2828#53782

Event Timeline

ardumont created this task.
ardumont renamed this task from Install webapp counter in the staging webapp/storage to Install webapp counters in the staging webapp/storage.Nov 9 2020, 10:32 AM
ardumont raised the priority of this task from Low to Normal.Dec 2 2020, 9:40 AM
ardumont updated the task description. (Show Details)

After T2828, It's more clear of what must be deployed to have the counters working on staging:

  • the counters can be intialized via the /stat/refresh endpoint of the storage api (Note: It will create more counters than production as directory_entry_* and revision_history are not counted in production)
  • Add a script/service to execute the `swh_update_counter_bucketed` in an infinite loop
  • Create the buckets in the object_counts_bucketed
    • per object type : identifier|bucket_start|bucket_end. value and last_update will be updated be the stored procedures.
  • configure prometheus sql exporter for db1.staging [1]
  • configure profile_exporter on pergamon
    • Update the script to ensure the data are filtered by environments (to avoid staging data to be included in production counts [2])
    • Configure a new cron
      • loading an empty file for historical data
      • creating a new export_file
  • update webapp to be able to configure the counter origin

[1] https://forge.softwareheritage.org/source/puppet-swh-site/browse/production/data/hostname/belvedere.internal.softwareheritage.org.yaml$1
[2] https://forge.softwareheritage.org/source/puppet-swh-site/browse/production/site-modules/profile/files/stats_exporter/export_archive_counters.py$39