LGTM

lgtm

The current serie were the counters are stored is named sql_swh_archive_object_count, the serie for swh-counters could be swh_archive_object_count

The 2 remaining disks were inserted in place of the 2 old sdd and sdf.
They needed to be configured in JBOD mode:

root@granet:~# megacli -PDMakeJBOD  -physdrv[32:3] -a0

A new vm counters0.internal.staging.swh.network is deployed and hosting redis, swh-counters and its journal-client.
The lag in staging will be recovered in a couple of hours.

To identify the disks to replace, the front led can be activated via the idrac interface.
The disks are :
scsi-35000c500ae759873 | /dev/sdd: Serial ZA1G3R1S -> Physical Disk 0:1:3
scsi-35000c500ae750b2f | /dev/sdf : Serial ZA1G3H81 -> Physical Disk 0:1:5

Removing the mirror mirror-2 with the disks scsi-35000c500ae750b2f and scsi-35000c500ae759873

root@granet:~# zpool remove hdd mirror-2
root@granet:~# zpool status hdd
  pool: hdd
 state: ONLINE
  scan: scrub repaired 0B in 0 days 17:39:22 with 0 errors on Sun Mar 14 18:03:23 2021
remove: Evacuation of mirror in progress since Fri Mar 19 10:45:22 2021
    1.03G copied out of 6.19T at 118M/s, 0.02% done, 15h18m to go
config:

And added to the hdd zfs pool:

• before

root@granet:~# zpool list
NAME   SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
hdd   29.1T  22.8T  6.29T        -         -    20%    78%  1.00x    ONLINE  -
ssd   10.3T  7.91T  2.44T        -         -    24%    76%  1.00x    ONLINE  -

• configuration

root@granet:~# ls -l /dev/disk/by-id | grep -e "wwn.*sdk" -e "wwn.*sdl"
lrwxrwxrwx 1 root root  9 Mar 19 10:10 wwn-0x5000c500cb46da4b -> ../../sdl
lrwxrwxrwx 1 root root  9 Mar 19 10:09 wwn-0x5000c500cb46e41b -> ../../sdk

root@granet:~# zpool add hdd mirror wwn-0x5000c500cb46da4b wwn-0x5000c500cb46e41b

root@granet:~# zpool status hdd
  pool: hdd
 state: ONLINE
  scan: scrub repaired 0B in 0 days 17:39:22 with 0 errors on Sun Mar 14 18:03:23 2021
config:

Disks configured in JBOD mode:

root@granet:~# megacli -PDMakeJBOD  -physdrv[32:10] -a0

ZFS status was reset and a scrub restart after the upgrade of the zfs packages.
No more errors are detected.

All the servers were updated. We took the opportunity to upgrade and restart them to apply the last updates.

plan for hypervisors / nodes upgrades:

• beaubourg relative
• workers[13..16]: stop services, upgrade package, shutdown, no needs to move them
• stop azure workers
• moma: upgrade packages, stop and restart on pompidou
• tate: upgrade packages, stop and restart on pompidou
• upgrade and stop somerset
• upgrade beaubourg and restart beaubourg
• restart somerset
• upgrade moma configuration to use somerset as database
• move back tate to beaubourg
• move back moma to beaubourg

• *esnode*
  • delaying node down detection and limit shard allocation to primaries

esnode2 ~ % export ES_NODE=192.168.100.62:9200                                              
esnode2 ~ % curl -XPUT -H "Content-Type: application/json" http://$ES_NODE/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

Plan:

• first the upgrade will be done on the elasticsearch server
• in parallel somerset can be updated
• after the webapp can be configured to use somerset as the principal database
• Upgrade of belveder
• upgrade of saam (with the help of @olasd)
• upgrade of belvedere
• and finally the kafka servers

branches unstable and buster configured
The builds passed and the package is available :

root@search0:~# apt search python3-swh.counters
Sorting... Done
Full Text Search... Done
python3-swh.counters/unknown 0.1.0-1+swh2~bpo10+1 all
  Software Heritage counters utilities

Build is working for unstable and buster.

fix pkg name

New repository created and configured: https://forge.softwareheritage.org/source/pytest-redis/