An improvment idea came to me during the refactoring, the script can be splitted and integrated in the 'swh-counters' codebase.

The proposal is:

• add an entry point that aggregate historical data from a json and the live data from prometheus and write it on disk. The historical json will be dropped by puppet
• add a crontab on the swh-counter vm that regularly call this new entrypoint
• add an entrypoint returning the content of the previously generated json file (or add an alias on the nginx). This entry point will be called by the webapp directly

Pros ordered :

• the code can be tested
• the complete pipeline can be deployed in the docker environment
• all the logic is on the swh-counters module and not shared with puppet
• The old and new versions can live together during the transition period

Cons:

• it adds a dependency between prometheus and swh-counters.

The cons could be addressed later if needed (by checking if the timeseries capabilities of redis can be used).

Great idea.

The final counters architecture looks like this with this improvment:

Source: https://archive.softwareheritage.org/swh:1:cnt:e9c2835742a4fa571e9fd1701dafbcfa945b663f;origin=https://forge.softwareheritage.org/source/snippets.git;visit=swh:1:snp:4c7110a28ae1b1b8c7e6881001ec9dfd75200920;anchor=swh:1:rev:cf77331a0f8f341f4b236e86dca616aadd661371;path=/vsellier/plantuml/T2912-counters/counters-with-archive.plantuml/

compared to https://hedgedoc.softwareheritage.org/UHZYNPrAS_q6LhgwT5UhoQ#target-proposal