Page MenuHomeSoftware Heritage
Create Task
Maniphest T4102

staging: Deploy swh-scrubber database and checkers
Closed, MigratedEdits Locked
Actions

Description

Plan:

  • D7494: Install scrubber (postgres) db
  • Check connection is ok
  • D7500: Update documentation on how to connect to db
  • Prepare git repository for debian package [1]
  • Debian package bootstrap
  • Fix debian build papercuts
    • Fix stuck python3-swh.graph.client build failure [2]
    • Fix python3-swh.scrubber (failing due to ^)
  • T4102#82560: Install schema in swh-scrubber db
  • T4102#82560: Grand read-only access to guest user
  • D7590, D7594: Deploy postgres checker service on a dedicated scrubber node

[1]

# First install the 'has-debian-packaging-branch' flag in repository in
# https://forge.softwareheritage.org/diffusion/edit/254/page/basics/
# Then install git pre-receive hook for ci to trigger the debian package build out of a git tag
root@tate:/srv/phabricator/hooks# phabricator-setup-hook /srv/phabricator/repos/254/ post-receive-swh-modules
Hook post-receive-swh-modules successfully installed on /srv/phabricator/repos/254/:

[2] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DGRPH/job/gbp-buildpackage/ back to green!

Revisions and Commits

rSPRE sysadm-provisioning
D7594rSPREc7878e9b2883 Provision scrubber0.internal.staging.swh.network
rSPSITE puppet-swh-site
D7596rSPSITEbbb04a2fbd21 Bump max connections for the postgresql server to 200
D7595rSPSITEd16a0fee4813 Deploy scrubber checks on remaining object types
D7595rSPSITE8f7183df87ef Bump swh-scrubber services memory usage
D7590rSPSITEc48217095bb7 Deploy scrubber postgres checker services to scrubber0.staging
D7494rSPSITEd0f7e9b9bf88 staging: Install new scrubber db in db1.staging
rDDOC Development documentation
D7500rDDOC3d53bd21da7f sysadm: Reference the swh-scrubber db

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Mar 28 2022, 1:51 PM
ardumont created this task.
vlorentz added a comment.EditedMar 28 2022, 2:00 PM

swh-scrubber will be made of the following components:

  1. the scrubber database (it's just a regular postgres database, accessed directly with psycopg2). It should remain under 100GB in the forseeable future.
  2. checkers, which each read one datastore (journal, storage but needs to connect directly to postgres, objstorage) and writes to the scrubber DB. They are all highly parallelizable (journal by spawning many identially processes; others by specifying a hash range to each process)
  3. fixers, which work on the scrubber database and no other datastore. One of them will also clone origins in temporary directories.
  4. reinjecters, which read fixed objects from the scrubber database and write them back to datastores

Let's focus on 1 and 2 for now, because 3 and 4 are WIPs, and running a checkers sweep is going to take longer than the entire dev time so we should start ASAP.

ardumont renamed this task from staging: Deploy swh-scrubber to staging: Deploy swh-scrubber database and checkers.Mar 28 2022, 3:29 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont added a project: System administration.Apr 1 2022, 4:51 PM
ardumont updated the task description. (Show Details)Apr 4 2022, 4:18 PM
ardumont added a revision: D7494: staging: Install new scrubber db in db1.staging.Apr 4 2022, 4:19 PM
ardumont changed the task status from Open to Work in Progress.Apr 4 2022, 5:56 PM
ardumont moved this task from Backlog to Weekly backlog on the System administration board.
ardumont moved this task from Weekly backlog to in-progress on the System administration board.
ardumont added a commit: rSPSITEd0f7e9b9bf88: staging: Install new scrubber db in db1.staging.Apr 5 2022, 10:00 AM
ardumont mentioned this in rSPPRIVCe2cc5f860ba2: staging: Install new scrubber db in db1.staging.Apr 5 2022, 10:04 AM
ardumont updated the task description. (Show Details)Apr 5 2022, 10:06 AM
ardumont updated the task description. (Show Details)
ardumont added a revision: D7500: sysadm: Reference the swh-scrubber db.Apr 5 2022, 10:20 AM
ardumont updated the task description. (Show Details)Apr 5 2022, 10:21 AM
ardumont added a commit: rDDOC3d53bd21da7f: sysadm: Reference the swh-scrubber db.Apr 5 2022, 4:58 PM
ardumont updated the task description. (Show Details)Apr 6 2022, 3:45 PM
ardumont mentioned this in rDSCRUB3afca0a1f34b: Import upstream version 0.0.1.Apr 6 2022, 3:49 PM
ardumont mentioned this in rDSCRUB208cad6ca686: Updated debian directory for version 0.0.1.
ardumont updated the task description. (Show Details)Apr 6 2022, 3:49 PM
ardumont updated the task description. (Show Details)Apr 6 2022, 4:13 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)Apr 6 2022, 4:16 PM
ardumont updated the task description. (Show Details)Apr 6 2022, 4:20 PM
ardumont updated the task description. (Show Details)
ardumont updated the task description. (Show Details)
ardumont added a comment.Apr 6 2022, 4:39 PM

Db got configured [1] with the following snippet [2]:

[1]

16:34:57 swh-scrubber@db1:5432=> \d
                   List of relations
+--------+------------------+----------+--------------+
| Schema |       Name       |   Type   |    Owner     |
+--------+------------------+----------+--------------+
| public | corrupt_object   | table    | swh-scrubber |
| public | datastore        | table    | swh-scrubber |
| public | datastore_id_seq | sequence | swh-scrubber |
| public | dbmodule         | table    | swh-scrubber |
| public | dbversion        | table    | swh-scrubber |
| public | object_origin    | table    | swh-scrubber |
+--------+------------------+----------+--------------+
(6 rows)
16:34:58 swh-scrubber@db1:5432=> grant select on all tables in schema public to guest;
GRANT
Time: 111.580 ms

[2]

$ setup-db.sh
+ DBPORT=5432
+ DBPASSWORD=<redacted>
+ DBUSER=swh-scrubber
+ DBNAME=swh-scrubber
+ DBHOST=db1.internal.staging.swh.network
+ CONNSTRING=postgresql://swh-scrubber:<redacted>@db1.internal.staging.swh.network:5432/swh-scrubber
+ swh db init-admin --dbname postgresql://swh-scrubber:<redacted>@db1.internal.staging.swh.network:5432/swh-scrubber scrubber
+ swh db init --dbname postgresql://swh-scrubber:<redacted>@db1.internal.staging.swh.network:5432/swh-scrubber scrubber
psql:/home/tony/work/inria/repo/swh/swh-environment/swh-core/swh/core/db/sql/36-dbmodule.sql:12: NOTICE:  relation "dbmodule" already exists, skipping
INFO:swh.core.cli.db:Initializing database version to 1 from the command line option --initial-version
DONE database for scrubber initialized at version 1

source: P1330

ardumont updated the task description. (Show Details)Apr 6 2022, 4:40 PM
ardumont updated the task description. (Show Details)Apr 7 2022, 3:39 PM
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Apr 13 2022, 5:15 PM
ardumont moved this task from deployed/landed/monitoring to in-progress on the System administration board.
ardumont mentioned this in rSPPRIVC6e7fc72e30a2: Reference censored sentry api token for swh-scrubber related services.Apr 15 2022, 10:49 AM
ardumont added a revision: D7590: Deploy swh-scrubber-checker-services to scrubber0.staging.Apr 15 2022, 12:34 PM
ardumont updated the task description. (Show Details)Apr 15 2022, 4:43 PM
ardumont moved this task from in-progress to code-review/await-feedback/pause on the System administration board.
ardumont added a commit: rSPSITEc48217095bb7: Deploy scrubber postgres checker services to scrubber0.staging.Apr 19 2022, 1:39 PM
ardumont added a revision: D7594: Provision scrubber0.internal.staging.swh.network.Apr 19 2022, 1:55 PM
ardumont updated the task description. (Show Details)Apr 19 2022, 1:56 PM
ardumont mentioned this in rSPSITE81d71fd370f4: Fix scrubber service start.Apr 19 2022, 2:19 PM
ardumont mentioned this in rSENV46df0422e3e4: watch-vagrant-conf: Fix manifests folder rsync issue.
ardumont mentioned this in rSENV8c963dcc607e: Vagrantfile: Reference the staging scrubber0 vm.
ardumont mentioned this in rSENV0eb28a0b0bee: Sync puppet master facts.
ardumont added a revision: D7595: Deploy scrubber checks on all object types and bump memory usage for those services.Apr 19 2022, 2:52 PM
ardumont moved this task from code-review/await-feedback/pause to deployed/landed/monitoring on the System administration board.Apr 19 2022, 2:52 PM
ardumont updated the task description. (Show Details)
ardumont added a commit: rSPREc7878e9b2883: Provision scrubber0.internal.staging.swh.network.Apr 19 2022, 2:54 PM
ardumont added a commit: rSPSITE8f7183df87ef: Bump swh-scrubber services memory usage.Apr 19 2022, 2:58 PM
ardumont added a commit: rSPSITEd16a0fee4813: Deploy scrubber checks on remaining object types.
ardumont mentioned this in rSPSITEfd8b64dc009f: Fix swh-scrubber services to check snapshot.Apr 19 2022, 3:02 PM
ardumont added a revision: D7596: Bump max connections for the postgresql server to 200.Apr 19 2022, 3:33 PM
ardumont added a commit: rSPSITEbbb04a2fbd21: Bump max connections for the postgresql server to 200.Apr 19 2022, 3:36 PM
ardumont mentioned this in rSENV2fe0c6ef441d: import-puppet-module: Make script more resilient when failure occurs.Apr 20 2022, 2:19 PM
ardumont closed this task as Resolved.Apr 20 2022, 2:24 PM
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
vlorentz mentioned this in T4324: production: Deploy swh-scrubber database and checkers.Jun 10 2022, 11:54 AM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:06 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal