• just keep the topic configuration as the journal split is not needed anymore
• fix the type in the commit message

I hesitated to do it, but as it should not move anymore now everything is reactivated, I choose to keep it as it.
We'll see if there are mouvements on this list in the near future

• version 0.7.0 release with the last improvement (D5576) of vlorentz (thanks)
• deployment done in staging
• the person counting has started on the live messages:

root@counters0:~# redis-cli
127.0.0.1:6379> pfcount person
(integer) 7

• now let reset the consumer offsets for the release and revision topics to backfill the person counter:

# offsets backup
/opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets --all-topics --to-current --dry-run  --export --group swh.counters.journal_client 2>&1 > ~/counters_journal_client_offsets.csv
# revision reset
 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.revision
# release reset
 /opt/kafka/bin/kafka-consumer-groups.sh --bootstrap-server $SERVER --reset-offsets  --group swh.counters.journal_client --to-earliest --execute --topic swh.journal.objects.release

the vm is configured and the new database schema on the staging database created.
psql is also configured with several services:

• swh-mirror : read-only connection on the swh-mirror schema
• admin-swh-mirror: r/w connection
• swh: read-only connection on the archive database (staging)

In D5576#141670, @vlorentz wrote:

In D5576#141650, @vsellier wrote:

I suppose the message.value() is returning a copy of the content

It does not: https://github.com/confluentinc/confluent-kafka-python/blob/b7f8dce998ec254f54c15e514850d7404c6a71a3/src/confluent_kafka/src/confluent_kafka.c#L439-L445

(Py_INCREF only increments the reference counter)

limit the pre-configured databases to the main database + swh-mirror

I suppose this implementation will be less effective in term of memory consumption as we will keep a copy of the message contents on the dict (I suppose the message.value() is returning a copy of the content)
It also makes the module aware of how the objects are serialized on kafka, which looks quite low level.

VM created by terraform :

mirror-tests_summary = 
hostname: mirror-test
fqdn: mirror-test.internal.staging.swh.network
network: ip=192.168.130.160/24,gw=192.168.130.1 macaddrs=E6:3C:8A:B7:26:5D

VM declared on the inventory : https://inventory.internal.softwareheritage.org/virtualization/virtual-machines/103/
Future ip will be 192.168.130.160

thanks for the diff you will propose. I will land this one in the interval.

It's for performance considerations only, for most of the counters, counting the keys is enough as it's the unique identifier in kafka.
The KeyOrientedJournalClient[1] is bypassing the object deserialization when a message is received, so a more classical client is needed for this specific Person case.

add missing doc strings

Update according the reviews' feedbacks

puppet ressources cleaned:

root@pergamon:~# /usr/local/sbin/swh-puppet-master-decomission clearly-defined.internal.staging.swh.network
+ puppet node deactivate clearly-defined.internal.staging.swh.network
Submitted 'deactivate node' for clearly-defined.internal.staging.swh.network with UUID 26eb9a73-add9-4745-b068-6106ab2b20b4
+ puppet node clean clearly-defined.internal.staging.swh.network
Notice: Revoked certificate with serial 256
Notice: Removing file Puppet::SSL::Certificate clearly-defined.internal.staging.swh.network at '/var/lib/puppet/ssl/ca/signed/clearly-defined.internal.staging.swh.network.pem'
clearly-defined.internal.staging.swh.network
+ puppet cert clean clearly-defined.internal.staging.swh.network
Warning: `puppet cert` is deprecated and will be removed in a future release.
   (location: /usr/lib/ruby/vendor_ruby/puppet/application.rb:370:in `run')
Notice: Revoked certificate with serial 256
+ systemctl restart apache2

rebase

• vm destroyed
• configuration removed for terraform
• database schemas cleared:
  • before:

root@db1:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
data  27.3T   623G  26.7T        -         -    16%     2%  1.00x    ONLINE  -

lgtm

The 2 disks were removed from the server and packaged to be sent to seagate.

The order was received and confirmed by dell ETA: 28th may
The detail was sent on the sysadm mailing list