After solving the problem the upgrade was pretty smooth. The firewall perform the following steps:

• upgrade to the last current minor version of the current major branch
• upgrade to the first minor version of the next major branch
• upgrade to the last minor version ot the current major branch

lgtm

Before starting the upgrade, we discovered 2 problem we had to fix:

1. The backup had no access to internet we block the upgrade
2. The master/backup switch was not working for 4 of the 8 VIPs

Is the user not used for the creation of the pgpass file ?

In T3190#61917, @ardumont wrote:

An improvment of the journal client is necessary to add the support of this configuration like for the producer:

Do you need such improvment though? According to the code you linked, you could pass a
producer_config dict with that key and value.

The journal client supports dynamic configuration via kwargs so no there is no need to improve it.

It seems the problem is not present anymore with a higher max message size ('500 * 1024 * 1024').

for the record, increasing the property message.max.bytes to 100 * 1024 * 1024 in the consumer configuration is not solving the problem

The same problem occured during the poc, theses messages were ignored by using this consumer configuration "errors.tolerance": 'all' [1].
I will try to find if there is a more elegant way to deal with this issue ;)

After talking with @rdicosmo, we finally chose to replace on each server the 4 HDD 2.4To by 6 SSD 1.9To to be sure we will have good performances and enought space for the future.
The quote wil nowl be sent to the purchasing service according to the usual procedure [1]

Final quotation sent for approval.
The details are:
3 PowerEdge R6515 (1u) with per server:

• 10 disks enclosure
• BOSS controller with 2 240Go cards (for system)
• 4 SAS 2.5" 10k 2.4To disks
• SFP+ network card
• 2 SFP cables
• 2 power supplies with their cables
• IDRac enterprise
• Rack mount rails with cable management

lgtm

credentials sent by PM

• unprivileged user :

username=swh-douardda
password=XXXXX

rebase

The final counters architecture looks like this with this improvment:

An improvment idea came to me during the refactoring, the script can be splitted and integrated in the 'swh-counters' codebase.

lgtm

lgtm

LGTM

node counters1.internal.softwareheritage.org deployed by terraform. The inventory section is created accordingly[1].
The journal_client is running.

:)
thanks

The counters are now exposed throught a /metrics enpoint and ingested by prometheus.
They are well tagged per environment so we will be able to isolate the counters for each one:

LGTM

lgtm

The current serie were the counters are stored is named sql_swh_archive_object_count, the serie for swh-counters could be swh_archive_object_count

The 2 remaining disks were inserted in place of the 2 old sdd and sdf.
They needed to be configured in JBOD mode:

root@granet:~# megacli -PDMakeJBOD  -physdrv[32:3] -a0

A new vm counters0.internal.staging.swh.network is deployed and hosting redis, swh-counters and its journal-client.
The lag in staging will be recovered in a couple of hours.