Page MenuHomeSoftware Heritage
Create Task
Maniphest T4415

Improve indexer workers performance
Closed, MigratedEdits Locked
Actions

Description

We are regularly noticed about elevated load on indexer workers since we migrated to journal client.

Possible improvments:

  • D8167: Dedicate journal client services per node
  • D8169: Use tmpfs
  • failed: D8170: Activate swap on local disk
  • Activate swap another way

Revisions and Commits

rSPRE sysadm-provisioning
AbandonedD8168 azure indexer workers: Add swap on nodes when provisioning
rSPSITE puppet-swh-site
ClosedD8167 Decrease parallelism on indexer workers running journal clients
D8170rSPSITE580c7688b33b Add waagent profile to azure worker role
D8169rSPSITEdb1bce12c25d Add /tmp as tmpfs partition for indexer workers

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Aug 1 2022, 5:45 PM
ardumont created this task.
ardumont changed the task status from Open to Work in Progress.Aug 1 2022, 5:47 PM
ardumont added a revision: D8169: Add /tmp as tmpfs partition for indexer workers.
ardumont added a revision: D8168: azure indexer workers: Add swap on nodes when provisioning.
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont added a revision: D8167: Decrease parallelism on indexer workers running journal clients.
ardumont updated the task description. (Show Details)
ardumont added a revision: D8170: Add waagent profile to azure worker role.Aug 1 2022, 6:13 PM
ardumont added a commit: rSPSITEdb1bce12c25d: Add /tmp as tmpfs partition for indexer workers.Aug 2 2022, 10:59 AM
ardumont added a commit: rSPSITE580c7688b33b: Add waagent profile to azure worker role.
ardumont updated the task description. (Show Details)Aug 2 2022, 11:13 AM

I've landed and deployed the diffs to make /tmp a tmpfs partition and activated the swap (as swapfile) on their local disk /mnt.

root@pergamon:~# clush -b -w @indexer-workers mount | grep /tmp
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4k,nr_inodes=209715200)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4k,nr_inodes=209715200)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4k,nr_inodes=209715200)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4k,nr_inodes=209715200)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4k,nr_inodes=209715200)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,size=4k,nr_inodes=209715200)
root@pergamon:~# clush -b -w @indexer-workers grep SwapSize /etc/waagent.conf
---------------
indexer-worker[01-06].euwest.azure (6)
---------------
ResourceDisk.SwapSizeMB=14336
root@pergamon:~# clush -b -w @indexer-workers grep Swap /etc/waagent.conf
---------------
indexer-worker[01-06].euwest.azure (6)
---------------
ResourceDisk.EnableSwap=y
ResourceDisk.SwapSizeMB=14336
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Aug 2 2022, 11:15 AM
ardumont mentioned this in rSPSITE46f6568c1250: azure_euwest: Fix tmpfs::size with the right amount.Aug 2 2022, 11:23 AM
ardumont updated the task description. (Show Details)Aug 4 2022, 3:56 PM
ardumont added a comment.Sep 8 2022, 3:04 PM

It seems to no longer be a problem now.
So closing.

ardumont closed this task as Resolved.Sep 8 2022, 3:04 PM
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:07 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal