Page MenuHomeSoftware Heritage
Feed Advanced Search

Jan 8 2023

gitlab-migration closed T3841: regularly scrub all the data stores of swh as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 10:23 PM · Datastore Scrubber, meta-task, Roadmap 2022, Storage manager
gitlab-migration changed the status of Restricted Maniphest Task, a subtask of T3841: regularly scrub all the data stores of swh, from Resolved to Migrated.
Jan 8 2023, 10:04 PM · Datastore Scrubber, meta-task, Roadmap 2022, Storage manager
gitlab-migration closed T4684: Publish scrubber metrics and create grafana dashboard as Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 5:05 PM · Datastore Scrubber
gitlab-migration changed the status of T4527: scrubber: keep a state file for postgresql datastores from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:37 PM · Datastore Scrubber
gitlab-migration changed the status of T4454: Create a dashboard for the swh-scrubber metrics, a subtask of T4435: scrubber log verbosity puts a risk on ELK , from Resolved to Migrated.
Jan 8 2023, 4:37 PM · Datastore Scrubber, System administrators
gitlab-migration changed the status of T4454: Create a dashboard for the swh-scrubber metrics from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:37 PM · Datastore Scrubber, System administrators
gitlab-migration changed the status of T4435: scrubber log verbosity puts a risk on ELK from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:37 PM · Datastore Scrubber, System administrators
gitlab-migration changed the status of T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function from Resolved to Migrated.

This task has been migrated to GitLab.

Jan 8 2023, 4:37 PM · Datastore Scrubber
gitlab-migration changed the status of T4136: Add an "history completeness check", a subtask of T3841: regularly scrub all the data stores of swh, from Resolved to Migrated.
Jan 8 2023, 4:36 PM · Datastore Scrubber, meta-task, Roadmap 2022, Storage manager

Jan 2 2023

vlorentz renamed T4684: Publish scrubber metrics and create grafana dashboard from Create scrubber metrics and grafana dashboard to Publish scrubber metrics and create grafana dashboard.
Jan 2 2023, 9:08 AM · Datastore Scrubber
vlorentz renamed T4684: Publish scrubber metrics and create grafana dashboard from Create grafana dashboard for scrubber metrics to Create scrubber metrics and grafana dashboard.
Jan 2 2023, 9:07 AM · Datastore Scrubber

Nov 14 2022

vlorentz added a project to T4684: Publish scrubber metrics and create grafana dashboard: Datastore Scrubber.
Nov 14 2022, 4:22 PM · Datastore Scrubber

Oct 24 2022

vlorentz closed T4527: scrubber: keep a state file for postgresql datastores as Resolved.
Oct 24 2022, 9:34 AM · Datastore Scrubber

Oct 19 2022

gitlab-migration changed the status of T4639: Deploy swh-scrubber v0.1.1, a subtask of T4527: scrubber: keep a state file for postgresql datastores, from Resolved to Migrated.
Oct 19 2022, 6:08 PM · Datastore Scrubber
gitlab-migration changed the status of T4639: Deploy swh-scrubber v0.1.1 from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:08 PM · System administration, Datastore Scrubber
gitlab-migration changed the status of T4387: Scrubber processes getting killed by OOM killer from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:07 PM · System administration, Datastore Scrubber
gitlab-migration changed the status of T4371: Deploy swh-scrubber on all storage instances, a subtask of T3841: regularly scrub all the data stores of swh, from Resolved to Migrated.
Oct 19 2022, 6:07 PM · Datastore Scrubber, meta-task, Roadmap 2022, Storage manager
gitlab-migration changed the status of T4324: production: Deploy swh-scrubber database and checkers from Resolved to Migrated.

This task has been migrated to GitLab.

Oct 19 2022, 6:07 PM · System administration, Datastore Scrubber

Oct 18 2022

ardumont closed T4639: Deploy swh-scrubber v0.1.1 as Resolved.
Oct 18 2022, 4:58 PM · System administration, Datastore Scrubber
ardumont closed T4639: Deploy swh-scrubber v0.1.1, a subtask of T4527: scrubber: keep a state file for postgresql datastores, as Resolved.
Oct 18 2022, 4:58 PM · Datastore Scrubber
ardumont moved T4639: Deploy swh-scrubber v0.1.1 from Weekly backlog to deployed/landed/monitoring on the System administration board.
Oct 18 2022, 11:22 AM · System administration, Datastore Scrubber
ardumont updated the task description for T4639: Deploy swh-scrubber v0.1.1.
Oct 18 2022, 11:11 AM · System administration, Datastore Scrubber
ardumont updated the task description for T4639: Deploy swh-scrubber v0.1.1.
Oct 18 2022, 10:38 AM · System administration, Datastore Scrubber
ardumont moved T4639: Deploy swh-scrubber v0.1.1 from Backlog to Weekly backlog on the System administration board.
Oct 18 2022, 10:23 AM · System administration, Datastore Scrubber

Oct 17 2022

vlorentz updated the task description for T4639: Deploy swh-scrubber v0.1.1.
Oct 17 2022, 3:11 PM · System administration, Datastore Scrubber
vlorentz triaged T4639: Deploy swh-scrubber v0.1.1 as High priority.
Oct 17 2022, 3:10 PM · System administration, Datastore Scrubber

Oct 7 2022

vlorentz added a revision to T4527: scrubber: keep a state file for postgresql datastores: D8641: storage_checker: Do not re-check ranges already marked as checked.
Oct 7 2022, 3:42 PM · Datastore Scrubber

Oct 4 2022

vlorentz added revisions to T4527: scrubber: keep a state file for postgresql datastores: D8608: db: Add table checked_ranges, D8609: storage_checker: Notify database when ranges are fully checked.
Oct 4 2022, 2:19 PM · Datastore Scrubber

Oct 3 2022

olasd added a comment to T4527: scrubber: keep a state file for postgresql datastores.

I think I'll implement it as a table in the scrubber's DB, this will make it easier to query the current status of scrubbing and add it to the Grafana dashboard

Oct 3 2022, 11:09 AM · Datastore Scrubber
vlorentz claimed T4527: scrubber: keep a state file for postgresql datastores.

I think I'll implement it as a table in the scrubber's DB, this will make it easier to query the current status of scrubbing and add it to the Grafana dashboard

Oct 3 2022, 10:27 AM · Datastore Scrubber
vlorentz added a project to T3841: regularly scrub all the data stores of swh: Datastore Scrubber.
Oct 3 2022, 10:24 AM · Datastore Scrubber, meta-task, Roadmap 2022, Storage manager

Sep 28 2022

vlorentz closed T4454: Create a dashboard for the swh-scrubber metrics, a subtask of T4435: scrubber log verbosity puts a risk on ELK , as Resolved.
Sep 28 2022, 11:15 AM · Datastore Scrubber, System administrators
vlorentz closed T4454: Create a dashboard for the swh-scrubber metrics as Resolved.

https://grafana.softwareheritage.org/d/t0eircZ4k/vlorentz-scrubber-metrics

Sep 28 2022, 11:15 AM · Datastore Scrubber, System administrators

Sep 9 2022

olasd triaged T4527: scrubber: keep a state file for postgresql datastores as High priority.
Sep 9 2022, 6:49 PM · Datastore Scrubber

Aug 22 2022

vsellier triaged T4454: Create a dashboard for the swh-scrubber metrics as High priority.
Aug 22 2022, 3:54 PM · Datastore Scrubber, System administrators

Aug 18 2022

olasd closed T4435: scrubber log verbosity puts a risk on ELK as Resolved.
Aug 18 2022, 5:21 PM · Datastore Scrubber, System administrators
olasd added a comment to T4435: scrubber log verbosity puts a risk on ELK .

I've deployed the quiesced scrubbers (v0.1.0) on both staging and production.

Aug 18 2022, 5:21 PM · Datastore Scrubber, System administrators
olasd added a revision to T4435: scrubber log verbosity puts a risk on ELK : D8251: Replace verbose logging with statsd probes.
Aug 18 2022, 1:31 PM · Datastore Scrubber, System administrators

Aug 12 2022

vsellier renamed T4435: scrubber log verbosity puts a risk on ELK from scrubber log verbosity put a risk on ELK to scrubber log verbosity puts a risk on ELK .
Aug 12 2022, 12:53 PM · Datastore Scrubber, System administrators
vsellier triaged T4435: scrubber log verbosity puts a risk on ELK as High priority.
Aug 12 2022, 12:52 PM · Datastore Scrubber, System administrators

Aug 4 2022

ardumont closed T4387: Scrubber processes getting killed by OOM killer as Resolved.
Aug 4 2022, 3:55 PM · System administration, Datastore Scrubber
ardumont renamed T4387: Scrubber processes getting killed by OOM killer from scrubber process killed by OOM killer to Scrubber processes getting killed by OOM killer.
Aug 4 2022, 3:54 PM · System administration, Datastore Scrubber
ardumont added a comment to T4387: Scrubber processes getting killed by OOM killer.

I've ended dropping the ballooning for that node.
As i've deployed twice as much services as before to scrub somerset as well [1]

Aug 4 2022, 3:54 PM · System administration, Datastore Scrubber
ardumont added a comment to T4387: Scrubber processes getting killed by OOM killer.

Still happening:

[  +0.063819] Killed process 1216634 (swh) total-vm:262252kB, anon-rss:206288kB, file-rss:724kB, shmem-rss:0kB
[  +0.077492] oom_reaper: reaped process 1216634 (swh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[Jul27 05:29] journalbeat invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[  +0.000002] journalbeat cpuset=/ mems_allowed=0
[  +0.000023] CPU: 2 PID: 688 Comm: journalbeat Not tainted 4.19.0-21-amd64 #1 Debian 4.19.249-2
--
[  +0.000002] [1250623]   997 1250623     3349       70    65536        4             0 systemctl
[  +0.000001] Out of memory: Kill process 1199696 (swh) score 82 or sacrifice child
[  +0.063517] Killed process 1199696 (swh) total-vm:201224kB, anon-rss:47784kB, file-rss:0kB, shmem-rss:0kB
[  +0.066909] oom_reaper: reaped process 1199696 (swh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[Jul27 06:12] kworker/0:0: page allocation failure: order:0, mode:0x6310ca(GFP_HIGHUSER_MOVABLE|__GFP_NORETRY|__GFP_NOMEMALLOC), nodemask=(null)
[  +0.000002] kworker/0:0 cpuset=/ mems_allowed=0
[  +0.000011] CPU: 0 PID: 1244099 Comm: kworker/0:0 Not tainted 4.19.0-21-amd64 #1 Debian 4.19.249-2
--
[  +0.000001] [1257717]   997 1257717     1369       35    49152        0             0 sudo
[  +0.000005] Out of memory: Kill process 1081929 (swh) score 136 or sacrifice child
[  +0.102631] Killed process 1081929 (swh) total-vm:303484kB, anon-rss:178744kB, file-rss:4kB, shmem-rss:0kB
[  +0.132669] oom_reaper: reaped process 1081929 (swh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[Jul27 09:29] swh invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0
[  +0.000002] swh cpuset=/ mems_allowed=0
[  +0.000010] CPU: 2 PID: 1209826 Comm: swh Not tainted 4.19.0-21-amd64 #1 Debian 4.19.249-2
--
[  +0.000001] [1264290]     0 1264290     3544       63    65536        0             0 check_journal
[  +0.000001] Out of memory: Kill process 1112652 (swh) score 70 or sacrifice child
[  +0.060327] Killed process 1112652 (swh) total-vm:195560kB, anon-rss:20828kB, file-rss:436kB, shmem-rss:0kB
[  +0.066241] oom_reaper: reaped process 1112652 (swh), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
Aug 4 2022, 10:53 AM · System administration, Datastore Scrubber

Jul 11 2022

ardumont moved T4387: Scrubber processes getting killed by OOM killer from in-progress to deployed/landed/monitoring on the System administration board.
Jul 11 2022, 2:21 PM · System administration, Datastore Scrubber
ardumont changed the status of T4387: Scrubber processes getting killed by OOM killer from Open to Work in Progress.
Jul 11 2022, 2:21 PM · System administration, Datastore Scrubber
ardumont added a comment to T4387: Scrubber processes getting killed by OOM killer.

After upgrading packages, and reboot.
This seems to have increased a bit its memory to something more sensible.
Let's see if the problem disappears altogether now.

Jul 11 2022, 2:21 PM · System administration, Datastore Scrubber
ardumont triaged T4387: Scrubber processes getting killed by OOM killer as Normal priority.
Jul 11 2022, 10:50 AM · System administration, Datastore Scrubber
vsellier updated the task description for T4387: Scrubber processes getting killed by OOM killer.
Jul 11 2022, 9:47 AM · System administration, Datastore Scrubber
vsellier updated the task description for T4387: Scrubber processes getting killed by OOM killer.
Jul 11 2022, 9:46 AM · System administration, Datastore Scrubber
vsellier created T4387: Scrubber processes getting killed by OOM killer.
Jul 11 2022, 9:43 AM · System administration, Datastore Scrubber

Jun 30 2022

ardumont closed T4324: production: Deploy swh-scrubber database and checkers as Resolved.
Jun 30 2022, 2:36 PM · System administration, Datastore Scrubber

Jun 28 2022

ardumont added a comment to T4324: production: Deploy swh-scrubber database and checkers.

Seems the job is well on its way:

07:59:35 swh-scrubber@belvedere:5432=> select now(), count(*) from corrupt_object ;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-28 05:59:39.490737+00 | 76511 |
+-------------------------------+-------+
(1 row)
Jun 28 2022, 8:00 AM · System administration, Datastore Scrubber

Jun 27 2022

ardumont moved T4324: production: Deploy swh-scrubber database and checkers from in-progress to deployed/landed/monitoring on the System administration board.
Jun 27 2022, 5:33 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 27 2022, 5:33 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 27 2022, 5:29 PM · System administration, Datastore Scrubber
ardumont added a revision to T4324: production: Deploy swh-scrubber database and checkers: D8041: scrubber: Deactivate the unneeded objstorage configuration part.
Jun 27 2022, 4:57 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 27 2022, 3:06 PM · System administration, Datastore Scrubber
ardumont added a revision to T4324: production: Deploy swh-scrubber database and checkers: D8039: Install scrubber services on scrubber nodes.
Jun 27 2022, 3:02 PM · System administration, Datastore Scrubber
ardumont added a revision to T4324: production: Deploy swh-scrubber database and checkers: D8038: Provision scrubber1 for checker services.
Jun 27 2022, 2:59 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 27 2022, 2:38 PM · System administration, Datastore Scrubber
ardumont shifted T4324: production: Deploy swh-scrubber database and checkers from the Restricted Space space to the S1 Public space.
Jun 27 2022, 2:37 PM · System administration, Datastore Scrubber
ardumont shifted T4324: production: Deploy swh-scrubber database and checkers from the S1 Public space to the Restricted Space space.
Jun 27 2022, 2:28 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 27 2022, 2:28 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 27 2022, 2:27 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 27 2022, 2:26 PM · System administration, Datastore Scrubber

Jun 24 2022

ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 24 2022, 5:13 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 24 2022, 4:53 PM · System administration, Datastore Scrubber
ardumont added a revision to T4324: production: Deploy swh-scrubber database and checkers: D8035: sysadm: Reference the production swh-scrubber db access.
Jun 24 2022, 4:53 PM · System administration, Datastore Scrubber
ardumont changed the status of T4324: production: Deploy swh-scrubber database and checkers from Open to Work in Progress.
Jun 24 2022, 4:15 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 24 2022, 4:15 PM · System administration, Datastore Scrubber
ardumont added a revision to T4324: production: Deploy swh-scrubber database and checkers: D8034: Deploy production swh-scrubber db connection.
Jun 24 2022, 4:10 PM · System administration, Datastore Scrubber
ardumont updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 24 2022, 3:59 PM · System administration, Datastore Scrubber

Jun 15 2022

ardumont moved T4324: production: Deploy swh-scrubber database and checkers from Backlog to Weekly backlog on the System administration board.
Jun 15 2022, 4:08 PM · System administration, Datastore Scrubber

Jun 10 2022

vlorentz updated the task description for T4324: production: Deploy swh-scrubber database and checkers.
Jun 10 2022, 11:54 AM · System administration, Datastore Scrubber
vlorentz triaged T4324: production: Deploy swh-scrubber database and checkers as Normal priority.
Jun 10 2022, 11:53 AM · System administration, Datastore Scrubber

May 31 2022

douardda closed T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Resolved.
May 31 2022, 4:52 PM · Datastore Scrubber
douardda added a revision to T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function : D7926: tests: use stock pytest_postgresql factory function.
May 31 2022, 4:43 PM · Datastore Scrubber
douardda triaged T4288: Replace usage of swh.core's postgresql_fact by stock pytest_postgresql's factory function as Normal priority.
May 31 2022, 4:40 PM · Datastore Scrubber
douardda edited Description on Datastore Scrubber.
May 31 2022, 4:39 PM
douardda edited Description on Datastore Scrubber.
May 31 2022, 4:39 PM
douardda edited Description on Datastore Scrubber.
May 31 2022, 4:38 PM
douardda created Datastore Scrubber.
May 31 2022, 4:38 PM