To avoid increasing the already high i/o pressure on the ceph cluster, the swap on the workers can be disabled.
it will avoid peak read/write peaks when the workers are on low memory period.
Description
Description
Revisions and Commits
Revisions and Commits
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T3444 26/07/2021: Unstuck infrastructure outage then post-mortem | ||
| Migrated | gitlab-migration | T3474 Disable swap on workers |
Event Timeline
Comment Actions
For the production and staging workers, swap has been deactivated and the /etc/fstab
entry got commented:
root@worker14:~# swapoff -a root@worker14:~# grep swap /etc/fstab #/dev/mapper/worker14--vg-swap_1 none swap sw 0 0
production
|----------+------------------+-------------+------------+----------------------| | workers | swap activated? | swap status | run status | remarks | |----------+------------------+-------------+------------+----------------------| | worker01 | X | swapoff | running | loader/lister worker | | worker02 | X | swapoff | running | loader/lister worker | | worker03 | X | swapoff | running | loader/lister worker | | worker04 | X | swapoff | stopped | loader/lister worker | | worker05 | X | swapoff | stopped | loader/lister worker | | worker06 | X | swapoff | stopped | loader/lister worker | | worker07 | X | swapoff | stopped | loader/lister worker | | worker08 | X | swapoff | running | loader/lister worker | | worker09 | X | swapoff | running | loader/lister worker | | worker10 | X | swapoff | running | loader/lister worker | | worker11 | X | swapoff | stopped | loader/lister worker | | worker12 | X | swapoff | stopped | loader/lister worker | | worker13 | X | swapoff | running | loader/lister worker | | worker14 | X | swapoff | running | loader/lister worker | | worker15 | X | swapoff | stopped | loader/lister worker | | worker16 | X | swapoff | stopped | loader/lister worker | |----------+------------------+-------------+------------+----------------------|
staging
|---------+-------+-------------+------------+-----------------------------| | workers | swap? | swap status | run status | remarks | |---------+-------+-------------+------------+-----------------------------| | worker0 | X | | running | loader/lister/cooker worker | | worker1 | | swapoff | stopped | // | | worker2 | X | | stopped | // | | worker3 | | swapoff | running | indexer/cooker worker | |---------+-------+-------------+------------+-----------------------------|
Grafana "maintenance" tag about this manipulation installed as well.
Comment Actions
as expected, there is an increase of the number of oom killers on the workers [1]:
It tends to confirm we should at least decrease the concurrency of the git loader.
[1] https://grafana.softwareheritage.org/d/j_6mA_Gnk/workers-oom-killer?orgId=1&from=now-6h&to=now