Page MenuHomeSoftware Heritage

Upgrade dulwich on celery workers
Closed, ResolvedPublic

Description

By playing with the new 'Save code now' task report feature (T1976, admin credentials required, can be found under //operations/webapp/admin in pass) freshly deployed to production, I noticed a lot of requests for Gitlab based origins were failing (for intance https://gitlab.adullact.net, https://gitlab.inria.fr, https://git.unistra.fr, https://framagit.org even https://gitlab.com).
The error appears when the [[ https://www.dulwich.io/ | dulwhich ]] module tries to clone such a repository (dulwich.errors.NotGitRepository exception is raised).

Digging a little buit further using Kibana, we can see more than 20000 git loading tasks ends up with the same error on the last two month.

I know that error pretty well as I stumbled across a couple of months ago and I took some time to debug it. The issue is now fixed since version 0.19.11 of dulwich.

Considering the increasing popularity of Gitlab based forges, we should consider upgrading dulwich on our celery workers. Checking python3-dulwich version info, we can see the issue will be fixed when upgrading to buster. As we use stretch in production, we need to backport dulwich>=0.19.11.

Event Timeline

anlambert triaged this task as Normal priority.Sep 7 2019, 12:35 AM
anlambert created this task.
ardumont added a subscriber: ardumont.

And nice work on the investigation and the fix within dulwich ;)

olasd closed this task as Resolved.Sep 10 2019, 12:10 PM
olasd claimed this task.
olasd added a subscriber: olasd.

I've backported dulwich 0.19.13-1 to our stretch repo, upgraded all workers and they're restarting.