@ardumont, looking at debug logs, the error in [1] is due to a permission issue

svnrdump: E170013: Unable to connect to a repository at URL 'https://llvm.org/svn/llvm-project'
svnrdump: E175013: Access to '/svn/llvm-project' forbidden

Build is green

Build has FAILED

Only drop dead code

@anlambert worse case scenario, if we don't find the source of the problem, we may want to add a fallback to the basic implem which tries and load messages one at a time.
Because then at least, we'd at least ingest something up to the point where we could no longer do.

Build is green

Build is green

Build is green

Build is green

Build is green

rebase

rebase

rebase

rebase

rebase

I forgot to mention there is a gift from dell on the server: an additional 600Go 10rpm disk

The server is installed. It remains few task to perform manually:

• configure the zfs datasets (will configure 2 mirror pool for ~12To available, tell me if it's not what it's expected)
• build few missing packages for bullseye (relative to the monitoring: prometheus-rabbitmq-exporter, prometheus-statsd-exporter, journalbeat)
• configure a rabbitmq admin user

I'll try to make multiple commits to drop progressively stuff so renames appear.

I made our jenkins ci build the patched dulwich with the fix discussed here.
It's currently uploaded in the swh debian repository [1]
I've deployed this on staging workers and trigger another run to
ensure everything is fine with it (again). If it is, i'll deploy on other workers tomorrow.

lgtm

yes pgbouncer will be used and it's configured by default to 2000 // connections
I don't know the kind of load the provenance client will generate but the default 100 connections allowed by postgres will be probably too short and needed to be increased too

In T3487#71230, @vsellier wrote:

• postgresql:13
  • 1000 parallel connections allowed

Build is green

Build is green

Build is green

Build is green

rebase

rebase

fix commit message

fix commit message

Build is green

Build is green

Build is green

As see with @aeviso , we will install the following components on the server (the os will be debian11)

• rabbitmq
• postgresql:13
  • a default swh-storage database will be managed by puppet
  • 1000 parallel connections allowed
  • shared_buffers 50go
• docker

Thanks!

rebase

rebase

split

Might be possible to simplify this a bit using a similar approach to https://forge.softwareheritage.org/source/swh-storage/browse/master/swh/storage/metrics.py$16-26

Overall looks ok to me but: