Latest dulwich release (0.20.43) finally drops caching of responses when using the HTTP client (I helped debugging the last remaining issue, see commit).
Current dulwich version we are using in production caches HTTP responses twice in memory which can be problematic when receiving a large packfile.
So we should upgrade dulwich in production to increase git loader performance in terms of CPU and memory usage.
I checked that the swh.loader.git tests are green with the new dulwich version.
The new package version has been uploaded to our debian package archive.
Now we "just" need to deploy it to staging and prod, and restart the relevant workers.
fwiw, jenkins is python3-dulwich aware.
I have this locally prior to see you claimed the task (so i 'll stop there without doing anything else ;)
... Status: successful Version: 0.20.43-1
I just fetched the upstream changes, merged the upstream/unstable into the debian/unstable-swh branch (fix the conflict) and gbp build packaged it and it worked out of the box ;).
As far as i remember, that's all that we need to do (well and then sign, tag, push to the forge repo)?
I don't see the point of that for packages that can be backported with no changes, which is what I had done before, so I admit I hadn't even looked.
I have this locally prior to see you claimed the task (so i 'll stop there without doing anything else ;)
... Status: successful Version: 0.20.43-1
That looks wrong, this is the version number for the unstable package, surely our local package should have a different (lower) version.
I just fetched the upstream changes, merged the upstream/unstable into the debian/unstable-swh branch (fix the conflict) and gbp build packaged it and it worked out of the box ;).
As far as i remember, that's all that we need to do (well and then sign, tag, push to the forge repo)?
Jelmer updates the Debian package of dulwich at every release, so there's no point having a separate "unstable-swh" branch (it would only be useful if we were packaging newer upstream versions than that available in Debian unstable). We should just branch off of the latest debian tag and merge that into the debian/buster-swh branch.
The simplest way to do it is, of course, to do:
apt-get source dulwich cd dulwich-* dch --local ~~swh~bpo10+ --distribution buster-swh --force-distribution 'Rebuild for buster-swh' sbuild -As --extra-repository 'deb [trusted=yes] https://debian.softwareheritage.org buster-swh main' --force-orig-source --no-clean-source --extra-repository 'deb http://deb.debian.org/debian buster-backports main' --build-dep-resolver aptitude debsign ../*.changes dcmd rsync ../*.changes pergamon:<...> ssh pergamon 'umask 0022; reprepro <...>'
fwiw, jenkins is python3-dulwich aware.
I don't see the point of that for packages that can be backported with no changes, which is what I had done before, so I admit I hadn't even looked.
It was meant as a comment prior to you commenting (when i saw you took the task ;)
...
Status: successful
Version: 0.20.43-1That looks wrong, this is the version number for the unstable package, surely our
local package should have a different (lower) version.
yes, it is wrong but that was not the point, the point was it's working as-is.
And yes, at that point, it is wrong (because i did not touch the debian/changelog yet).
I stopped since i saw you took the task (and then delayed my comment for some reason
because i'm side-track-y today).
I just fetched the upstream changes, merged the upstream/unstable into the debian/unstable-swh branch (fix the conflict) and gbp build packaged it and it worked out of the box ;).
As far as i remember, that's all that we need to do (well and then sign, tag, push to the forge repo)?Jelmer updates the Debian package of dulwich at every release, so there's no point having a separate "unstable-swh" branch (it would only be useful
Yes, but he does not release always consistently. It has been useful in the past. Once,
at least for one patch (merged upstream but not released) yet.
if we were packaging newer upstream versions than that available in Debian unstable).
We should just branch off of the latest debian tag and merge that into the
debian/buster-swh branch.
ack
The simplest way to do it is, of course, to do:
apt-get source dulwich cd dulwich-* dch --local ~~swh~bpo10+ --distribution buster-swh --force-distribution 'Rebuild for buster-swh' sbuild -As --extra-repository 'deb [trusted=yes] https://debian.softwareheritage.org buster-swh main' --force-orig-source --no-clean-source --extra-repository 'deb http://deb.debian.org/debian buster-backports main' --build-dep-resolver aptitude debsign ../*.changes dcmd rsync ../*.changes pergamon:<...> ssh pergamon 'umask 0022; reprepro <...>'
Yes, i'm using some internal script which wraps those (well based out of our ci
job/docs)
Anyway, thanks.
I was curious about how you do it.
I've restarted the staging workers (loader_git and loader_high_priority) with the new dulwich version