Grafana postgresql overview: https://grafana.softwareheritage.org/d/PEKz-Ygiz/postgresql-server-overview

(god I hate how noisy the puppet coding style makes diffs)

From what I can tell, in your make_app_from_configfile function, the conf object is never used, only checked. That feels very brittle to me and I think the checks should happen in the swh.deposit.settings.production module directly.

After some more stewing and discussion with @zack, we'll be going for the "upgrade to pg 11 and restart replication from scratch" route;

Looks fine.

At that point I guess the storage_archiver worker should just be called archiver (the name dates back from the point the archiver was still in the storage namespace), but as it's not in use currently I don't think it's really worth bothering.

Looks fine to me considering ::profile::swh::deploy::worker::base is properly setup to clean up the /etc/softwareheritage/worker directory (as by itself, removing a resource definition does not remove the resource).

After reading some mailing list posts discussing the error message, and discussion with @ftigeot:

Logs on primary:

2019-02-19 14:27:44 UTC [15973]: [1-1] user=postgres,db=softwareheritage LOG:  starting logical decoding for slot "pgl_softwareheritage_prado_somerset"
2019-02-19 14:27:44 UTC [15973]: [2-1] user=postgres,db=softwareheritage DETAIL:  streaming transactions committing after 18607/9189A578, reading WAL from 18607/9189A578
2019-02-19 14:27:44 UTC [15973]: [3-1] user=postgres,db=softwareheritage ERROR:  record with incorrect prev-link 5403A/2E2F1829 at 18607/9189A578 
2019-02-19 14:27:44 UTC [15973]: [4-1] user=postgres,db=softwareheritage LOG:  could not receive data from client: Connection reset by peer

I'm slightly reordering what you wrote here, sorry!

I've pushed an updated kombu to our repository.

Thanks for recording this task; I'll use the opportunity to document the reasoning behind the current internal networking setup, to try and make sure nothing is forgotten before migrating it.

Some related work : https://fosdem.org/2019/schedule/event/small_files_in_swift_cluster/

I've documented the extra hoops people need to jump through for git push to work on https://wiki.softwareheritage.org/index.php?title=Code_review_in_Phabricator&type=revision&diff=964&oldid=899

I've moved the repositories that need it to have their staging area set to https.

In T1522#27987, @ardumont wrote:

use https for the staging area configuration (which will allow people to push it even if nothing is configured, by just typing their password)

That'd mean also to change that uri to be read/write, it's read-only for now.

One more possibility is to document that people need to setup an ssh key on their account to be able to push their changes.

During the pvmove off of /dev/md3, the root filesystem for uffizi ended up being remounted r/o. I've shut it down, fsck'd it, and booted it back up.

In D1099#23381, @ardumont wrote:

sounds good

from where does the jobExists function come from though? (is that coming from some jenkins setup for example).

Fix copyright year

Rebase

That's a fairly large repo (as seen with how the content bundles get spread out to limit their size). It looks like it has some large directories (e.g. the .bugs directory looks like it has a lot of entries) so I'm not too surprised.

Reference T1460 in the commit

Re-re-rebase

Re-rebase

Rebasé

Add comment + rebase

Rebase

Add all the comments + rebase