The issue was not visible on an Azure VM but a Proxmox one.

Content of $facts['os'] on the problematic host:

os => {"name"=>"Debian", "family"=>"Debian", "release"=>{"major"=>"9", "minor"=>"5", "full"=>"9.5"}, "lsb"=>{"distcodename"=>"stretch", "distid"=>"Debian", "distdescription"=>"Debian GNU/Linux 9.5 (stretch)", "distrelease"=>"9.5", "majdistrelease"=>"9", "minordistrelease"=>"5"}}

It is caused by this piece of code in site-modules/profile/manifests/prometheus/apt_config.pp

if $facts['os']['distro']['codename'] == 'stretch' {

Now, this configuration works perfectly on my workstation running logstash-6.3.2-1 but for some reason, the existing logstash instance running on banco.internal.softwareheritage.org is unable to filter Apache logs.

Logstash filter configuration

Logstash output.conf configuration

Filebeat configuration on moma:/etc/filebeat/filebeat.yml

Journalbeat installed on moma in order to ship some Apache logs to the logstash instance on banco.
Logstash configuration changed to put raw Apache logs in apache_logs-$date indexes.

Some logstash indexes cannot be reindexed.
Part of the Elasticsearch error message:

"mapper_parsing_exception","reason":"failed to parse [swh_logging_args_return_value]","caused_by":{"type":"illegal_argument_exception","reason":"Failed to parse value [None] as only [true] or [false] are allowed."}}

The D11 and DS11 VM sizes are not very interesting.
Among the rest, the existing VM can only be resized to an e2s_v3 model.

All remaining non-swh-worker logs deleted from legacy logstash-* indexes.

Fixed by @olasd in https://forge.softwareheritage.org/rSPREd529d3b26a01a6643cf6eed2920a25b08317e571#change-6Ckng2kSodym .

Public IP address deassociated from dbreplica1 and then deleted via the Azure portal interface today.

Possible Azure VM types with 14-16GB RAM available on the Euwest zone:

• a2mv2 16GB RAM $90.52/month
• E2v3 16GB RAM $116.80/month
• E2sv3 16GB RAM $116.80/month
• D11v2 14GB RAM $138.70/month
• DS11v2 14GB RAM $138.70/month

dbreplica1.euwest.azure.internal.softwareheritage.org runs a slave instance of the softwareheritage-indexer database, with its master running on somerset.internal.softwareheritage.org.

https://webapp0.softwareheritage.org/browse/search/ does not answer quickly when entering a new keyword for the first time.
Tried "danana" => got answers after more than one minute.
Also tried "zarg" => the wheel stopped spinning after one minute, with no explanation. A text message saying the pattern was not found could have been helpful.

It seems like deleting old documents takes a heavy toll on the cluster.
So far, for every month of old logstash indexes cleaned, at least one node member started to misbehave and had to be restarted after excessive timeouts and/or other issues including constant garbage collection and disk trashing.

Even though all delete requests were previously successfully processed, non-swh-workers data remain in the legacy logstash-* indexes.
This is not an entirely unexpected behavior. It is possible resource limitations prevented the old Banco node from processing all deletion requests in a bounded time frame.
Deletion queries will be rerun index by index in this way:

 curl -i -H'Content-Type: application/json' \
     -XPOST "http://esnode2.internal.softwareheritage.org:9200/logstash-2018.02.31/_delete_by_query?pretty=true" -d '
{
	"query" : { "bool" : { "must_not" : [{ "match" : { "systemd_unit" : "swh-worker@" }}] }}
}'

The swh_workers-2018.03.07 index contained non-swh-workers documents and was cleaned this way:

curl -i -H'Content-Type: application/json' \
     -XPOST "http://esnode3.internal.softwareheritage.org:9200/swh_workers-2018.03.07/_delete_by_query?pretty=true" -d '
{
	"query" : { "bool" : { "must_not" : [{ "match" : { "systemd_unit" : "swh-worker@" }}] }}
}'

Possible solutions may include:

• First installing a dummy nginx configuration file
• Change Debian's service startup policy with `/usr/sbin/policy-rc.d```
• Stop using Varnish locally and replace it with Azure caching services

Some logstash-xxx indexes appear to still contain non-swh_workers related information.
Blindly reindexing them according to this issue description will not be enough.

The cluster now has three brand new nodes; all existing data has been copied and the original Banco node removed.
Issue technically fixed but left open in order to track related Elasticsearch requests.