great ;)

The save code now queue statistics are now displayed on the status.io page[1] as an example. The data are refreshed each 5 minutes.

Metrics can easily be pushed to the status page.
The simple poc for the save code now request is available here : https://forge.softwareheritage.org/source/snippets/browse/master/sysadmin/status.io/update_metrics.py

for the status.swh.org point of view, status.io is providing some api endpoint to push metrics. It should be possible to add some metrics (up to 10 with our plan) to expose the behavior of the platform (daily/weekly and monthly statistics).
As a first step, we could expose the number of pending save code now requests and the number of origin visits to have some live data. An example of a status page with metrics : https://status.docker.com/
I'm working on a code snippet to test the integration feasibility/complexity.

> I also recall now that vincent added a graph [1] recently enough.

This to try and compare a bit the counter approaches together.

So that's still using the old plumbing at least for that part.

[1] https://grafana.softwareheritage.org/goto/BlkwHorMz

In T2912#64208, @ardumont wrote:

What about the old counter pipeline? Has it been decommissioned already?

I don't think so as I do not recall seeing diffs about clean up.

In any case, it's not part of what's currently deployed (so no risk for
data mangling if that's part the concern).

I'll keep it open until the docker env is ok as well (see the diff D5615).

Deployed (production/staging)

What about the old counter pipeline? Has it been decommissioned already?

In T2912#64174, @ardumont wrote:

Last bits deployed on archive.s.o (including the author counters).

Last bits deployed on archive.s.o (including the author counters).

Hear hear, it's kept up now:

ardumont@counters1:~% date;redis-cli pfcount person
Sat 24 Apr 2021 05:31:18 PM UTC
(integer) 42190221

and the authors are now displayed on staging and production (webapp1)

The lag for the production can be followed here: https://grafana.softwareheritage.org/goto/Di2H3z9Gk
(staging has already recovered)

the swh-counters is deployed in production too:

• upgrade swh-counters package and restart swh-counters backend and journal

root@counters1:~# apt dist-upgrade
...
Setting up python3-swh.counters (0.7.0-1~swh1~bpo10+1) ...
root@counters1:~# systemctl stop swh-counters-journal-client.service 
root@counters1:~# systemctl restart gunicorn-swh-counters.service 
root@counters1:~# systemctl start swh-counters-journal-client.service 
root@counters1:~# redis-cli pfcount person
(integer) 7

The count of the person already starts

• stopping the journal-client to be able to reset the releases and revisions offsets

root@counters1:~# systemctl stop swh-counters-journal-client.service

• reset the offsets

vsellier@kafka1 ~ % /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run  --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets.csv
# revision reset
vsellier@kafka1 ~ % 
 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.revision
# release reset
vsellier@kafka1 ~ %  /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.release 
# checks
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run  --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets-backfill.csv 
vsellier@kafka1 ~ % diff ~/counters_journal_client_offsets.csv ~/counters_journal_client_offsets-backfill.csv | less 
1c1
< "swh.journal.objects.revision",25,8275180
---
> "swh.journal.objects.revision",25,0
8c8
< "swh.journal.objects.release",128,78484
---
> "swh.journal.objects.release",128,0
16c16
...

• journal client restarted

root@counters1:~# systemctl start swh-counters-journal-client.service

• the person counters is growing fastly

root@counters1:~# date;redis-cli pfcount person
Fri 23 Apr 2021 10:55:54 AM UTC
(integer) 72358
root@counters1:~# date;redis-cli pfcount person
Fri 23 Apr 2021 10:55:57 AM UTC
(integer) 80618

Also [1] to follow through the journal client consumption (it has data now ;)

I think you can close D5573 which is obsolete now with the latest change.

• version 0.7.0 release with the last improvement (D5576) of vlorentz (thanks)
• deployment done in staging
• the person counting has started on the live messages:

root@counters0:~# redis-cli
127.0.0.1:6379> pfcount person
(integer) 7

• now let reset the consumer offsets for the release and revision topics to backfill the person counter:

# offsets backup
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run  --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets.csv
# revision reset
 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.revision
# release reset
 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.release

All checks green both for production/staging and hg/svn/git,

Nevertheless, that error should have been detected on the first loading so some adaptation is needed in the svn loader.

In any case, independently from this, for the monitoring, I was set on modifying the
actual svn origin used to something else not hosted on github.

On a related note, it may be useful to regularly report requests that did not complete (either as success or failure) in a reasonable amount of time after being scheduled.

Thanks for the heads up ;)

The error is related to that github origin erroneously submitted with a svn visit type.

There is no hg origin for now but this is configurable so as soon as one one is found, this will be checked as well.

In the mean time, this error should not be too blocking on pergamon.

Damn, I tried to deploy the end to end save code now check but computer says no [1]

don't forget to count committers too

Let's go for it, then. May you take this over?

In T2912#63245, @vsellier wrote:

This kind of journal client will be necessary in any case if we want to extend the usage of the counters for other perimeters (metadata count, origin per forge, ...)

I would really like to keep the author counter: how complex is it to add it?

In T2912#63242, @vsellier wrote:

Staging webapp[1] and webapp1 on production [2] are now configured to use swh-counters to display the historical values and the live object counts.

Staging webapp[1] and webapp1 on production [2] are now configured to use swh-counters to display the historical values and the live object counts.

Deployment done on staging and production. The new counters are currently only activated on webapp1

• version 0.1.296(swh2) released and deployed on all the webapp

swh-web v0.0.295 is released and deployed on staging and production.

The puppet script generating the aggragated data is updated and was run to refresh the data.
The webapp can be released with this diff now

one additional point before releasing this, the puppet script making the aggregation need to be improved as it only merge the data for the content graph :
https://forge.softwareheritage.org/source/puppet-swh-site/browse/production/site-modules/profile/files/stats_exporter/export_archive_counters.py$109

The P1005 convert the data added by the webapp to json data that can be added to the /usr/local/share/swh-data/history-counters.munin.json file.
This content can be added on the file before the change on the webapp is released. it will just add few duplicate points to render, but with no effect on the final rendering

great news!