Page MenuHomeSoftware Heritage

Generate historical data from the new counters series
Closed, ResolvedPublic

Description

The script[1] caching the historical data needs to be updated to generate the data from the new series.
It can be improved to add some parameters to allow to specify:

  • the time serie to use
  • the list of collection to generate
  • the interval (can be useful to reduce the number of points to render on the webapp)

[1] https://forge.softwareheritage.org/source/puppet-swh-site/browse/production/site-modules/profile/files/stats_exporter/export_archive_counters.py

Event Timeline

vsellier triaged this task as Normal priority.Mar 22 2021, 6:31 PM
vsellier created this task.
vsellier removed a project: Web app.
vsellier removed subscribers: rdicosmo, olasd, vlorentz and 4 others.
vsellier changed the task status from Open to Work in Progress.Thu, Mar 25, 2:38 PM

An improvment idea came to me during the refactoring, the script can be splitted and integrated in the 'swh-counters' codebase.

The proposal is:

  • add an entry point that aggregate historical data from a json and the live data from prometheus and write it on disk. The historical json will be dropped by puppet
  • add a crontab on the swh-counter vm that regularly call this new entrypoint
  • add an entrypoint returning the content of the previously generated json file (or add an alias on the nginx). This entry point will be called by the webapp directly

Pros ordered :

  • the code can be tested
  • the complete pipeline can be deployed in the docker environment
  • all the logic is on the swh-counters module and not shared with puppet
  • The old and new versions can live together during the transition period

Cons:

  • it adds a dependency between prometheus and swh-counters.

The cons could be addressed later if needed (by checking if the timeseries capabilities of redis can be used).

vsellier reopened this task as Work in Progress.Wed, Apr 7, 5:42 PM

Reopening as the release is not working on the stable branch

Everything is released correctly and deployed on staging