Page MenuHomeSoftware Heritage

stats_exporter: Refactor and add docstrings to graph data export script
AcceptedPublic

Authored by ardumont on Sat, Aug 10, 6:03 PM.

Details

Reviewers
ftigeot
vlorentz
Group Reviewers
Reviewers
Summary

Now that the script is ok, i thought it was the right time to improve it a bit.

So here it goes:

  • refactor a bit the script
  • Add docstrings and comments to clarify some part
  • simplify the data computation (no if, simply inline the call for history data solely for content).
  • Make the script parametric to reuse the setup from the manifest (avoiding future problems, hard-coding values in the long run is not what we want)

This also:

  • export-stats: Rename script with consistent _ for python filename
  • extract variables from puppet manifest to avoid duplication
  • export_archive_counters: Update required python dependencies (requests was not mentioned)
Test Plan

prior version script run.
current version script run.
Check output are the same and it is.

$ python ./export-archive_counters.py > test.origin.json
$ python ./export_archive_counters.py --history-data-file ./history-counters.munin.json > test.json
$ stat test.json
  File: test.json
  Size: 85468           Blocks: 168        IO Block: 4096   regular file
Device: fe02h/65026d    Inode: 43916075    Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/    tony)   Gid: ( 1000/    tony)
Access: 2019-08-10 23:32:28.555767531 +0200
Modify: 2019-08-10 23:31:41.050503170 +0200
Change: 2019-08-10 23:31:41.050503170 +0200
 Birth: -
$ diff test.json test.origin.json  # <- no diff
$ sha512sum test.json
147670ea7b4d8d388adfcf09586e5c143113d26f6eee265ba2d0b88185b3809b6429668c3ba96a46829a68149a2355ff069e8018935eebc2ebd923069e2f17bb  test.json
$ sha512sum test.origin.json
147670ea7b4d8d388adfcf09586e5c143113d26f6eee265ba2d0b88185b3809b6429668c3ba96a46829a68149a2355ff069e8018935eebc2ebd923069e2f17bb  test.origin.json

Diff Detail

Repository
rSPSITE puppet-swh-site
Branch
staging
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 7220
Build 10196: arc lint + arc unit

Event Timeline

ardumont created this revision.Sat, Aug 10, 6:03 PM
ardumont updated this revision to Diff 6192.Sat, Aug 10, 11:33 PM
  • script: Make the script parametric
  • export: Make the command reuse the manifest's parameters
  • export_archive_counters: Update required python dependencies
ardumont edited the test plan for this revision. (Show Details)Sat, Aug 10, 11:36 PM
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the summary of this revision. (Show Details)
ardumont retitled this revision from stats_exporter: Refactor and add docstrings to export script to stats_exporter: Refactor and add docstrings to graph data export script.Sat, Aug 10, 11:38 PM
vlorentz accepted this revision.Mon, Aug 12, 1:25 PM
vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
site-modules/profile/files/stats_exporter/export_archive_counters.py
78–79

Nitpick:

result = [adapt_format(i)
          for i in data['data']['result'][0]['values']]
92

I don't think there should be a default.

site-modules/profile/manifests/export_archive_counters.pp
25

Is this the right place for a data file? (although it's immutable)

This revision is now accepted and ready to land.Mon, Aug 12, 1:25 PM
ardumont edited the test plan for this revision. (Show Details)Mon, Aug 12, 4:50 PM