Within the deposit logs, we can see lots of prometheus metrics query noise [1]:
Dec 11 09:59:06 deposit python3[3678588]: 2020-12-11 09:59:06 [3678588] django.security.DisallowedHost:ERROR Invalid HTTP_HOST header: 'webapp.staging.swh.network:443'. You may need to add 'webapp.staging.swh.network' to ALLOWED_HOSTS. Dec 11 09:59:06 deposit python3[3678588]: 2020-12-11 09:59:06 [3678588] django.request:WARNING Bad Request: /metrics/prometheus Dec 11 09:59:06 deposit python3[3678588]: 2020-12-11 09:59:06 [3678588] gunicorn.access:INFO 127.0.0.1 - - [11/Dec/2020:09:59:06 +0000] "GET /metrics/prometheus HTTP/1.1" 400 26 "-" "Prometheus/2.7.1+ds"
After investigation, this is due to the fallback mechanism of vhosts in
varnish. As no query matches the vhost selection, neither webapp nor deposit
matches (due to the extra :443), those queries end up being forwarded to the
default vhost, the deposit. Most probably determined by lexicographic order on
the file name.
Note that it currently works in production because the webapp there is not
named webapp (after "deposit" in lexicographic order) but archive (so
before the deposit).
A fix is currently being cooked so that the match work properly and the
requests are forwarded to the proper server.