Differential D8381

cluster-archive-staging: Activate thanos sidecar service
ClosedPublic
Actions

Authored by ardumont on Sep 2 2022, 9:33 AM.

Details

Reviewers

vsellier

Group Reviewers

System administrators

Maniphest Tasks

T4144: Elastic worker infrastructure

Commits

rSPRE7620b1f22dde: cluster-archive-staging: Activate thanos sidecar service

Summary

To push metrics into an azure objstore bucket (the same way other swh thanos services
are configured).

Next step is to configure our thanos gateway service to query that bucket (cf.
thanos.internal.admin.swh.network node).

Note: This required the objstore config to be pushed as secret in the cluster (this has
been done in k8s-private-data, cf. comment)

Related to T4144

Test Plan

Deployed yesterday and this has pushed new metrics in the bucket
metrics-sesi-rocquencourt-rancher-staging-0.

I did not attend to historical data to 1. avoid pushing duplicates in the bucket now
that new data is in and 2. because it's not willing to work too much [1].

It must be possible to do nonethessless but unnecessary for now (unless we want to add
extra labels on the data, cf. commented code)

[1] P1444

Diff Detail

Repository

rSPRE sysadm-provisioning

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Sep 2 2022, 9:33 AM

Herald added a reviewer: System administrators. · View Herald TranscriptSep 2 2022, 9:33 AM

ardumont requested review of this revision.Sep 2 2022, 9:33 AM

Harbormaster completed remote builds in B31281: Diff 30242.Sep 2 2022, 9:33 AM

vsellier accepted this revision.Sep 2 2022, 9:40 AM

This revision is now accepted and ready to land.Sep 2 2022, 9:40 AM

ardumont added inline comments.Sep 2 2022, 9:40 AM

proxmox/terraform/staging/cluster-archive-staging.tf
199	That's the mean to add extra label on the metrics prior to push those. That may be worth checking that a bit more though i think.
217	Those are the defaults but to avoid looking for those if we need some at some point, i have explicited those.

Add external labels to metrics

Harbormaster completed remote builds in B31283: Diff 30245.Sep 2 2022, 10:32 AM

Map properly external labels for thanos to take into account

(it's actually hard to troubleshoot... but there finally found it!)

out of the thanos-sidecar service running in the cluster:

level=info ts=2022-09-02T09:42:04.855304768Z caller=sidecar.go:155 msg="successfully loaded prometheus external labels" external_labels="{cluster_name=\"archive-staging\", domain=\"staging\", environment=\"staging\", infrastructure=\"kubernetes\", prometheus=\"cattle-monitoring-system/rancher-monitoring-prometheus\", prometheus_replica=\"prometheus-rancher-monitoring-prometheus-0\"}"

So now, remains to clean up that bucket and start from scratch (so it's properly
labelled)...

Harbormaster completed remote builds in B31290: Diff 30252.Sep 2 2022, 11:44 AM

Closed by commit rSPRE7620b1f22dde: cluster-archive-staging: Activate thanos sidecar service (authored by ardumont). · Explain WhySep 2 2022, 3:39 PM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rSPRE7620b1f22dde: cluster-archive-staging: Activate thanos sidecar service.

ardumont mentioned this in T4144: Elastic worker infrastructure.Sep 2 2022, 3:49 PM