Datastore Scrubber project.
Source code: swh-scrubber
Details
Details
Description
Jan 8 2023
Jan 8 2023
gitlab-migration changed the status of Restricted Maniphest Task, a subtask of T3841: regularly scrub all the data stores of swh, from Resolved to Migrated.
gitlab-migration changed the status of T4527: scrubber: keep a state file for postgresql datastores from Resolved to Migrated.
gitlab-migration changed the status of T4454: Create a dashboard for the swh-scrubber metrics, a subtask of T4435: scrubber log verbosity puts a risk on ELK , from Resolved to Migrated.
gitlab-migration changed the status of T4454: Create a dashboard for the swh-scrubber metrics from Resolved to Migrated.
gitlab-migration changed the status of T4435: scrubber log verbosity puts a risk on ELK from Resolved to Migrated.
gitlab-migration changed the status of T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function from Resolved to Migrated.
gitlab-migration changed the status of T4136: Add an "history completeness check", a subtask of T3841: regularly scrub all the data stores of swh, from Resolved to Migrated.
Jan 2 2023
Jan 2 2023
vlorentz renamed T4684: Publish scrubber metrics and create grafana dashboard from Create scrubber metrics and grafana dashboard to Publish scrubber metrics and create grafana dashboard.
vlorentz renamed T4684: Publish scrubber metrics and create grafana dashboard from Create grafana dashboard for scrubber metrics to Create scrubber metrics and grafana dashboard.
Nov 14 2022
Nov 14 2022
vlorentz added a project to T4684: Publish scrubber metrics and create grafana dashboard: Datastore Scrubber.
Oct 24 2022
Oct 24 2022
Oct 19 2022
Oct 19 2022
gitlab-migration changed the status of T4639: Deploy swh-scrubber v0.1.1, a subtask of T4527: scrubber: keep a state file for postgresql datastores, from Resolved to Migrated.
gitlab-migration changed the status of T4387: Scrubber processes getting killed by OOM killer from Resolved to Migrated.
gitlab-migration changed the status of T4371: Deploy swh-scrubber on all storage instances, a subtask of T3841: regularly scrub all the data stores of swh, from Resolved to Migrated.
gitlab-migration changed the status of T4324: production: Deploy swh-scrubber database and checkers from Resolved to Migrated.
Oct 18 2022
Oct 18 2022
ardumont closed T4639: Deploy swh-scrubber v0.1.1, a subtask of T4527: scrubber: keep a state file for postgresql datastores, as Resolved.
ardumont moved T4639: Deploy swh-scrubber v0.1.1 from Weekly backlog to deployed/landed/monitoring on the System administration board.
ardumont updated the task description for T4639: Deploy swh-scrubber v0.1.1.
ardumont updated the task description for T4639: Deploy swh-scrubber v0.1.1.
ardumont moved T4639: Deploy swh-scrubber v0.1.1 from Backlog to Weekly backlog on the System administration board.
Oct 17 2022
Oct 17 2022
vlorentz updated the task description for T4639: Deploy swh-scrubber v0.1.1.
Oct 7 2022
Oct 7 2022
Oct 4 2022
Oct 4 2022
Oct 3 2022
Oct 3 2022
olasd added a comment to T4527: scrubber: keep a state file for postgresql datastores.
I think I'll implement it as a table in the scrubber's DB, this will make it easier to query the current status of scrubbing and add it to the Grafana dashboard
Sep 28 2022
Sep 28 2022
vlorentz closed T4454: Create a dashboard for the swh-scrubber metrics, a subtask of T4435: scrubber log verbosity puts a risk on ELK , as Resolved.
Sep 9 2022
Sep 9 2022
Aug 22 2022
Aug 22 2022
Aug 18 2022
Aug 18 2022
olasd added a comment to T4435: scrubber log verbosity puts a risk on ELK .
I've deployed the quiesced scrubbers (v0.1.0) on both staging and production.
Aug 12 2022
Aug 12 2022
vsellier renamed T4435: scrubber log verbosity puts a risk on ELK from scrubber log verbosity put a risk on ELK to scrubber log verbosity puts a risk on ELK .
Aug 4 2022
Aug 4 2022
ardumont renamed T4387: Scrubber processes getting killed by OOM killer from scrubber process killed by OOM killer to Scrubber processes getting killed by OOM killer.
ardumont added a comment to T4387: Scrubber processes getting killed by OOM killer.
I've ended dropping the ballooning for that node.
As i've deployed twice as much services as before to scrub somerset as well [1]
ardumont added a comment to T4387: Scrubber processes getting killed by OOM killer.
Still happening:
[ +0.063819] Killed process 1216634 (swh) total-vm:262252kB, anon-rss:206288kB, file-rss:724kB, shmem-rss:0kB [ +0.077492] oom_reaper: reaped process 1216634 (swh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [Jul27 05:29] journalbeat invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 [ +0.000002] journalbeat cpuset=/ mems_allowed=0 [ +0.000023] CPU: 2 PID: 688 Comm: journalbeat Not tainted 4.19.0-21-amd64 #1 Debian 4.19.249-2 -- [ +0.000002] [1250623] 997 1250623 3349 70 65536 4 0 systemctl [ +0.000001] Out of memory: Kill process 1199696 (swh) score 82 or sacrifice child [ +0.063517] Killed process 1199696 (swh) total-vm:201224kB, anon-rss:47784kB, file-rss:0kB, shmem-rss:0kB [ +0.066909] oom_reaper: reaped process 1199696 (swh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [Jul27 06:12] kworker/0:0: page allocation failure: order:0, mode:0x6310ca(GFP_HIGHUSER_MOVABLE|__GFP_NORETRY|__GFP_NOMEMALLOC), nodemask=(null) [ +0.000002] kworker/0:0 cpuset=/ mems_allowed=0 [ +0.000011] CPU: 0 PID: 1244099 Comm: kworker/0:0 Not tainted 4.19.0-21-amd64 #1 Debian 4.19.249-2 -- [ +0.000001] [1257717] 997 1257717 1369 35 49152 0 0 sudo [ +0.000005] Out of memory: Kill process 1081929 (swh) score 136 or sacrifice child [ +0.102631] Killed process 1081929 (swh) total-vm:303484kB, anon-rss:178744kB, file-rss:4kB, shmem-rss:0kB [ +0.132669] oom_reaper: reaped process 1081929 (swh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB [Jul27 09:29] swh invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 [ +0.000002] swh cpuset=/ mems_allowed=0 [ +0.000010] CPU: 2 PID: 1209826 Comm: swh Not tainted 4.19.0-21-amd64 #1 Debian 4.19.249-2 -- [ +0.000001] [1264290] 0 1264290 3544 63 65536 0 0 check_journal [ +0.000001] Out of memory: Kill process 1112652 (swh) score 70 or sacrifice child [ +0.060327] Killed process 1112652 (swh) total-vm:195560kB, anon-rss:20828kB, file-rss:436kB, shmem-rss:0kB [ +0.066241] oom_reaper: reaped process 1112652 (swh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Jul 11 2022
Jul 11 2022
ardumont moved T4387: Scrubber processes getting killed by OOM killer from in-progress to deployed/landed/monitoring on the System administration board.
ardumont changed the status of T4387: Scrubber processes getting killed by OOM killer from Open to Work in Progress.
ardumont added a comment to T4387: Scrubber processes getting killed by OOM killer.
After upgrading packages, and reboot.
This seems to have increased a bit its memory to something more sensible.
Let's see if the problem disappears altogether now.
vsellier updated the task description for T4387: Scrubber processes getting killed by OOM killer.
vsellier updated the task description for T4387: Scrubber processes getting killed by OOM killer.