Page MenuHomeSoftware Heritage

pergamon takes a long time to apply its own manifest
Closed, MigratedEdits Locked

Description

around 5min [1] even though its overall system seems not that loaded [3] (and the changes to apply are very close to nothing [2])

[1]

root@pergamon:~# puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for pergamon.softwareheritage.org
Info: Applying configuration version '1642085546'
Notice: /Stage[main]/Profile::Systemd_journal::Journalbeat/File[/var/lib/journalbeat]/owner: owner changed 118 to 'root'
Notice: /Stage[main]/Profile::Systemd_journal::Journalbeat/File[/var/lib/journalbeat]/group: group changed 'nogroup' to 'root'
Notice: /Stage[main]/Profile::Bind_server::Primary/Resource_record[admin-db1/CNAME]/ensure: created
Notice: Applied catalog in 321.34 seconds

[2]

root@pergamon:~# puppet agent --test
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for pergamon.softwareheritage.org
Info: Applying configuration version '1642086083'
Notice: Applied catalog in 307.25 seconds

[3] https://grafana.softwareheritage.org/goto/5rd0scJ7z?orgId=1

Event Timeline

ardumont triaged this task as Normal priority.Jan 13 2022, 4:00 PM
ardumont created this task.
ardumont updated the task description. (Show Details)
olasd claimed this task.
olasd added a subscriber: olasd.

Looks like it's been spending a lot of time on DNS lookups.

I had made bind9 only run on ipv4 for it to stop spamming logs with failed lookups to ipv6 nameservers. However this makes puppet's use of dig @localhost .... to check dns records very slow, because it waits for a second to fallback to ipv4 from the default ipv6 localhost.

Restoring bind9 listening to both ipv4 and ipv6 restored puppet runs to a more reasonable 50 seconds.