swh-search0

• stopping writes

root@search0:~# systemctl stop swh-search-journal-client@objects
root@search0:~# systemctl stop swh-search-journal-client@indexed
root@search0:~# puppet agent --disable "zfs upgrade"
``
- package upgrades
- `swh-search0` rebooted
- `swh-search0` rebooted
- all service are up and running

• All workers and journal clients stopped before upgrading storage1 and db1

sudo clush -b -w @staging-workers 'puppet agent --disable "zfs upgrade"; cd /etc/systemd/system/multi-user.target.wants; for unit in swh-worker@*; do systemctl disable $unit; done; systemctl stop swh-worker@*'

storage1

• package upgrade done
• restart done without any problem
• all ser

db1

• package upgrade done
• restart done without any problem

• All workers reactivated

sudo clush -b -w @staging-workers 'systemctl default'
sudo clush -b -w @staging-workers 'puppet agent --enable; puppet agent --test'
sudo clush -b -w @staging-workers 'cd /etc/systemd/system/multi-user.target.wants; for unit in swh-worker@*; do systemctl start $unit; done'

• journal client on search0 reactivated

It seems everything is ok. There is a lot of timeout on some database queries, but according to sentry to problem was already there before the upgrade:
https://sentry.softwareheritage.org/share/issue/dbf854e3362742bd924d5a0418b3ee00/

The production upgrade will be performed next week

Plan:

• first the upgrade will be done on the elasticsearch server
• in parallel somerset can be updated
• after the webapp can be configured to use somerset as the principal database
• Upgrade of belveder
• upgrade of saam (with the help of @olasd)
• upgrade of belvedere
• and finally the kafka servers

• *esnode*
  • delaying node down detection and limit shard allocation to primaries

esnode2 ~ % export ES_NODE=192.168.100.62:9200                                              
esnode2 ~ % curl -XPUT -H "Content-Type: application/json" http://$ES_NODE/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": "primaries"
  }
}'

{"acknowledged":true,"persistent":{"cluster":{"routing":{"allocation":{"enable":"primaries"}}}},"transient":{}}%

• package update
• reboot
• reactivate allocation

curl -XPUT -H "Content-Type: application/json" http://$ES_NODE/_cluster/settings -d '{
  "persistent": {
    "cluster.routing.allocation.enable": null
  }
}'

• wait for a green cluster and restart for the next node

plan for hypervisors / nodes upgrades:

• beaubourg relative
• workers[13..16]: stop services, upgrade package, shutdown, no needs to move them
• stop azure workers
• moma: upgrade packages, stop and restart on pompidou
• tate: upgrade packages, stop and restart on pompidou
• upgrade and stop somerset
• upgrade beaubourg and restart beaubourg
• restart somerset
• upgrade moma configuration to use somerset as database
• move back tate to beaubourg
• move back moma to beaubourg

• branly relative
• pompidou: upgrade, stop and restart on uffizi
• logstash0: upgrade, stop and restart on uffizi
• saatchi: move on hypervisor3 (to avoid to stop all the workers)
• louvre: move to hypervisor3
• thyssen: upgrade, stop and restart on uffizi
• riverside: upgrade stop and restart on hypervisor3
• rp1: upgrade stop and restart on hypervisor3 (hedgedoc downtime)
• bardo: upgrade stop and restart on hypervisor3 (Hedgedoc downtime)
• kelvingrove: upgrade stop, restart on hypervisor3 (keycloak downtime)
• boatbucket: upgrade stop restart on uffizi (and let it there ?)
• [be continued...]

• saam / belvedere:
• (take the opportunity to upgrade and restart saatchi)
• upgrade and restart saam
• upgrade and restart belvedere
• restart all the workers