Maniphest T4435

scrubber log verbosity puts a risk on ELK
Closed, MigratedEdits Locked
Actions

Assigned To

gitlab-migration

Authored By

	vsellier
	Aug 12 2022, 12:52 PM

Tags

Subscribers

gitlab-migration

Description

An alert is raised by the monitoring for the free disk space of the ELK nodes.

The log producers repartition for the last week:

It starts when the scrunner v0.0.6 was deployed

Revisions and Commits

rDSCRUB Datastore Scrubber
	D8251	rDSCRUBc6f711ecbfb5 Replace verbose logging with statsd probes

Related Objects
Search...

		Status	Assigned	Task
		Migrated	gitlab-migration	T4435 scrubber log verbosity puts a risk on ELK
		Migrated	gitlab-migration	T4454 Create a dashboard for the swh-scrubber metrics

Event Timeline

vsellier renamed this task from scrubber log verbosity put a risk on ELK to scrubber log verbosity puts a risk on ELK .Aug 12 2022, 12:52 PM

vsellier triaged this task as High priority.

vsellier created this task.

olasd added a revision: D8251: Replace verbose logging with statsd probes.Aug 18 2022, 1:31 PM

olasd added a commit: rDSCRUBc6f711ecbfb5: Replace verbose logging with statsd probes.Aug 18 2022, 2:33 PM

I've deployed the quiesced scrubbers (v0.1.0) on both staging and production.

Someone™ should turn the new metrics into dashboards: https://grafana.softwareheritage.org/explore?orgId=1&left=%7B%22datasource%22:%22000000002%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22editorMode%22:%22code%22,%22expr%22:%22sum%20by%20%28object_type,instance,scrubber_instance%29%20%28rate%28swh_scrubber_objects_hashed_total%7Binstance%21%3D%5C%22jenkins-debian1.internal.softwareheritage.org%5C%22%7D%5B5m%5D%29%29%22,%22legendFormat%22:%22__auto%22,%22range%22:true,%22instant%22:true%7D%5D,%22range%22:%7B%22from%22:%22now-24h%22,%22to%22:%22now%22%7D%7D

olasd closed this task as Resolved.Aug 18 2022, 5:21 PM

olasd claimed this task.

vlorentz closed subtask T4454: Create a dashboard for the swh-scrubber metrics as Resolved.Sep 28 2022, 11:15 AM

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T4454: Create a dashboard for the swh-scrubber metrics from Resolved to Migrated.Jan 8 2023, 4:37 PM