Page MenuHomeSoftware Heritage
Paste P1104

What if workers played with kernel linux forks at the same time... and with also huge repositories
ActivePublic

Authored by ardumont on Jul 28 2021, 12:19 PM.
root@pergamon:~# clush -b -w @swh-workers "systemctl status swh-worker@loader_git" | grep -i linux
Jul 28 09:22:30 worker01 python3[116142]: [2021-07-28 09:22:30,504: INFO/ForkPoolWorker-400] Load origin 'https://git.launchpad.net/~kerneltoast/+git/bionic-linux-hwe' with type 'git'
Jul 28 10:05:11 worker02 python3[54928]: [2021-07-28 10:05:11,162: INFO/ForkPoolWorker-229] Load origin 'https://git.launchpad.net/~arighi/+git/xenial-linux-kvm' with type 'git'
dulwich.errors.GitProtocolError: unexpected http resp 401 for https://git.launchpad.net/~arighi/+git/xenial-linux-kvm/info/refs?service=git-upload-pack
swh.loader.exception.NotFound: unexpected http resp 401 for https://git.launchpad.net/~arighi/+git/xenial-linux-kvm/info/refs?service=git-upload-pack
Jul 28 10:05:11 worker02 python3[54928]: [2021-07-28 10:05:11,422: INFO/ForkPoolWorker-229] Load origin 'https://git.launchpad.net/~kamalmostafa/ubuntu/+source/linux-aws/+git/cosmic' with type 'git'
Jul 28 10:06:44 worker03 python3[62694]: [2021-07-28 10:06:44,384: INFO/ForkPoolWorker-252] Load origin 'https://git.launchpad.net/~ddstreet/+git/linux' with type 'git'
Jul 28 08:11:57 worker04 python3[49519]: [2021-07-28 08:11:57,065: INFO/ForkPoolWorker-216] Load origin 'https://git.launchpad.net/~connork/+git/disco-linux-raspi2' with type 'git'
Jul 28 08:17:00 worker04 python3[48836]: [2021-07-28 08:17:00,701: INFO/ForkPoolWorker-211] Load origin 'https://git.launchpad.net/~connork/+git/xenial-linux-gcp' with type 'git'
Jul 28 09:18:43 worker04 python3[39101]: [2021-07-28 09:18:43,230: INFO/ForkPoolWorker-82] Load origin 'https://git.launchpad.net/~kerneltoast/+git/bionic-linux-azure' with type 'git'
Jul 28 09:29:26 worker05 python3[26459]: [2021-07-28 09:29:26,997: INFO/ForkPoolWorker-42] Load origin 'https://git.launchpad.net/~p-pisati/+git/eoan-linux-azure' with type 'git'
Jul 28 08:33:13 worker06 python3[48595]: [2021-07-28 08:33:13,624: INFO/ForkPoolWorker-279] Load origin 'https://git.launchpad.net/~manjo/+git/disco-linux-snapdragon' with type 'git'
Jul 28 08:11:04 worker07 python3[46693]: [2021-07-28 08:11:04,635: INFO/ForkPoolWorker-235] Load origin 'https://git.launchpad.net/~sforshee/+git/linux-unstable' with type 'git'
Jul 28 08:14:26 worker07 python3[47641]: [2021-07-28 08:14:26,608: INFO/ForkPoolWorker-238] Load origin 'https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-gcp/+git/cosmic' with type 'git'
Jul 28 08:19:31 worker07 python3[48461]: [2021-07-28 08:19:31,476: INFO/ForkPoolWorker-247] Load origin 'https://git.launchpad.net/~canonical-kernel/ubuntu/+source/linux-oracle/+git/cosmic' with type 'git'
Jul 28 10:17:23 worker08 python3[47084]: [2021-07-28 10:17:23,926: INFO/ForkPoolWorker-288] Load origin 'https://git.launchpad.net/~tyhicks/ubuntu/+source/linux/+git/bionic' with type 'git'
Jul 28 10:20:35 worker10 python3[51552]: [2021-07-28 10:20:35,880: INFO/ForkPoolWorker-241] Load origin 'https://git.launchpad.net/~kerneltoast/+git/xenial-linux-gcp' with type 'git'
Jul 28 09:35:32 worker11 python3[48651]: [2021-07-28 09:35:32,944: INFO/ForkPoolWorker-266] Load origin 'https://git.launchpad.net/~p-pisati/+git/bionic-linux-azure' with type 'git'
Jul 28 10:19:00 worker12 python3[46584]: [2021-07-28 10:19:00,644: INFO/ForkPoolWorker-285] Load origin 'https://git.launchpad.net/~kerneltoast/+git/bionic-linux-aws-5.0' with type 'git'
Jul 28 08:28:31 worker13 python3[112471]: [2021-07-28 08:28:31,731: INFO/ForkPoolWorker-276] Load origin 'https://git.launchpad.net/~kerneltoast/+git/bionic-linux-oem-osp1' with type 'git'
Jul 28 08:14:28 worker14 python3[51088]: [2021-07-28 08:14:28,896: INFO/ForkPoolWorker-237] Load origin 'https://git.launchpad.net/~bladernr/ubuntu/+source/linux/+git/bionic' with type 'git'
Jul 28 08:50:20 worker15 python3[48199]: [2021-07-28 08:50:20,450: INFO/ForkPoolWorker-300] Load origin 'https://git.launchpad.net/~kmously/ubuntu/+source/linux/+git/eoan' with type 'git'
# And/Or on huge repositories that takes time to ingest
root@pergamon:~# clush -b -w @swh-workers "systemctl status swh-worker@loader_git" | grep "'eventful"
Jul 28 08:17:00 worker04 python3[48836]: [2021-07-28 08:17:00,652: INFO/ForkPoolWorker-211] Task swh.loader.git.tasks.UpdateGitRepository[2d582885-8245-44f2-b984-572a11d71a26] succeeded in 1069.010756263s: {'status': 'eventful'}
Jul 28 09:18:43 worker04 python3[39101]: [2021-07-28 09:18:43,163: INFO/ForkPoolWorker-82] Task swh.loader.git.tasks.UpdateGitRepository[a7466144-5f3e-4aeb-a2dc-c810c8691c3c] succeeded in 16790.309666889s: {'status': 'eventful'}
Jul 28 09:29:26 worker05 python3[26459]: [2021-07-28 09:29:26,895: INFO/ForkPoolWorker-42] Task swh.loader.git.tasks.UpdateGitRepository[cdd1ce87-1e25-4ead-8873-8f8c581f4bc9] succeeded in 28944.167442414997s: {'status': 'eventful'}
Jul 28 08:14:26 worker07 python3[47641]: [2021-07-28 08:14:26,539: INFO/ForkPoolWorker-238] Task swh.loader.git.tasks.UpdateGitRepository[70000a95-3fbe-4d56-adf4-35156475ec1c] succeeded in 574.7364137539989s: {'status': 'eventful'}
Jul 28 08:19:31 worker07 python3[48461]: [2021-07-28 08:19:31,410: INFO/ForkPoolWorker-247] Task swh.loader.git.tasks.UpdateGitRepository[e026099c-0463-4879-9d4d-17a83b0010f8] succeeded in 843.2227554689962s: {'status': 'eventful'}

Event Timeline

ardumont changed the title of this paste from What if all workers played with kernel linux forks at the same time... to What if workers played with kernel linux forks at the same time... and with also huge repositories.Jul 28 2021, 12:22 PM
ardumont edited the content of this paste. (Show Details)

So loader_git on big repositories implies lots of memory (so maybe swap is used as well), thus slowness on ingesting other origins...

Load flatlines seen in [1].

That loading seems to have subsided and everything is back to smoother load.

[1] https://grafana.softwareheritage.org/goto/IBAvKwWnz?orgId=1