namely add 2 metrics:
- swh_journal_client_handle_message_total: (counter) the total number of kafka messages that have been handled by the client,
- swh_journal_client_status: (gauge) report the current status of the kafka consumer (waiting or processing).
Since these metrics will be created by any kafka consumer (any swh.journal.client), tags should be added to the statsd metrics (eg. via the STATS_TAGS env var) to distinguish between the different types of consumers (e.g. hostname, consumer type/role, etc.)
Especially for the swh_journal_client_status to work properly
in an elastic execution environment (docker swarm, k8s, ...) it is
mandatory to add a consumer-specific tag (e.g. the hostname) and make
sure the TTL for the metrics in prometherus-statsd-exporter is pretty
short (e.g. 1h), so that restarted consumers are not kept for too long/forever by statsd-exporter.
Depends on D6884