In order to improve reindexation speed, replicas are initially disabled for new indexes.
Do not delete historical logstash-* indexes before being sure the new indexes have been reconfigured to use at least one replica per shard and the shards properly replicated.

Do we really care that much about reindexation speed? This reindexation is a one-shot, unattended process.

If we really do, could we make the replica-less template pattern narrower, to apply only to indexes for historical data? As the reindexation will likely take days, it'd be nice to avoid creating the new daily indexes with an unsafe configuration.

If we do it in batch and stop+reapply the regular template before midnight there shouldn't be any issue.
Templates are only used at index creation time.

Many logstash-$date indexes effectively have 0 documents left after the initial systemlog data deletion phase of T977 and can simply be deleted.

Some logstash-xxx indexes appear to still contain non-swh_workers related information.
Blindly reindexing them according to this issue description will not be enough.

Example of an Elasticsearch query showing non-swh_worker data:

curl -i -H'Content-Type: application/json' -XGET http://esnode1.internal.softwareheritage.org:9200/swh_workers-2018.03.07/_search -d '
{
	"query" : { "bool" : { "must_not" : [{ "match" : { "systemd_unit" : "swh-worker@" }}] }}
}'

Some logstash indexes cannot be reindexed.
Part of the Elasticsearch error message:

"mapper_parsing_exception","reason":"failed to parse [swh_logging_args_return_value]","caused_by":{"type":"illegal_argument_exception","reason":"Failed to parse value [None] as only [true] or [false] are allowed."}}

logstash appears to be a good substitute to the failing reindex Elasticsearch API.
It naturally skips documents whose contents appear to be invalid when sent to an Elasticsearch cluster.

Logstash invocation:

logstash -f logstash-reindex.conf

Configuration used:

input {
	elasticsearch {
		hosts => "http://esnode1.internal.softwareheritage.org:9200"
		index => "logstash-2017.04.19"
		query => '{ "query": { "match_all": {} } }'
		scroll => "30m"
		docinfo => true
	}
}

output {
	elasticsearch {
		hosts => "http://esnode2.internal.softwareheritage.org:9200"
		index => "swh_workers-2017.04.19"
		document_id => "%{[@metadata][_id]}"
	}
}