I've added an autossh-mmca.service unit on belvedere to have a reverse tunnel from localhost@mmca:5345 -> localhost@belvedere:5432.

I've granted mmca access to the production journal for user swh-provenance-mmca.

On mmca:

• I've upgraded swh.provenance in a new virtualenv.

• I've created systemd system units for the provenance revision journal clients on mmca:

swhprovenance@mmca:~$ ls -l /etc/systemd/system/swh-provenance-revision-client*
-rw-r--r-- 1 root root 701 Jul 22 13:48 /etc/systemd/system/swh-provenance-revision-client-debug@.service
-rw-r--r-- 1 root root 671 Jul 22 13:50 /etc/systemd/system/swh-provenance-revision-client@.service

I've started the debug unit for one of the clients, and the non-debug unit for 63 others.

To monitor the progress of the journal clients: https://grafana.softwareheritage.org/d/KvQqUhsWz/kafka-consumers-lag?orgId=1&var-environment=production&var-cluster=rocquencourt&var-consumer=swh-provenance-mmca-revision-0&from=now-3h&to=now&refresh=30m

To monitor the provenance progress: https://grafana.softwareheritage.org/d/rsm10b27z/provenance-metrics?orgId=1&from=now-7d&to=now is still valid.

The journal client has a bunch of already-processed revisions to skip over/throw away, so the actual provenance metrics will probably start picking back up in a few days.

FWIW I've also started the provenance storage server as a systemd system service now.