Page MenuHomeSoftware Heritage
Differential D7494

staging: Install new scrubber db in db1.staging
ClosedPublic
Actions

Authored by ardumont on Apr 4 2022, 4:19 PM.

Details

Reviewers
vsellier
Group Reviewers
System administrators
Maniphest Tasks
T4102: staging: Deploy swh-scrubber database and checkers
Commits
rSPSITEd0f7e9b9bf88: staging: Install new scrubber db in db1.staging
Summary

Related to T4102

Test Plan

Add the new scrubber db to db1.staging:

$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff --octocatalog-diff-args --no-truncate-details --to staging db1.internal.staging.swh.network
Found host db1.internal.staging.swh.network
...
diff origin/production/db1.internal.staging.swh.network current/db1.internal.staging.swh.network
*******************************************
  Concat::Fragment[_postgres] =>
   parameters =>
     content =>
      @@ -8,3 +8,4 @@
       swh-web = host=localhost port=5433 auth_user=postgres
       swh-mirror = host=localhost port=5433 auth_user=postgres
      +swh-scrubber = host=localhost port=5433 auth_user=postgres
      _
*******************************************
  Concat_fragment[_postgres] =>
   parameters =>
     content =>
      @@ -8,3 +8,4 @@
       swh-web = host=localhost port=5433 auth_user=postgres
       swh-mirror = host=localhost port=5433 auth_user=postgres
      +swh-scrubber = host=localhost port=5433 auth_user=postgres
      _
*******************************************
  Pgbouncer::Databases[pgbouncer_module_databases] =>
   parameters =>
     databases =>
      - [{"source_db"=>"swh", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh"}, {"source_db"=>"swh-scheduler", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-scheduler"}, {"source_db"=>"swh-vault", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-vault"}, {"source_db"=>"swh-lister", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-lister"}, {"source_db"=>"swh-deposit", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-deposit"}, {"source_db"=>"swh-indexer", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-indexer"}, {"source_db"=>"swh-web", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-web"}, {"source_db"=>"swh-mirror", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"swh-mirror"}]
      + [{"source_db"=>"swh", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh"}, {"source_db"=>"swh-scheduler", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-scheduler"}, {"source_db"=>"swh-vault", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-vault"}, {"source_db"=>"swh-lister", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-lister"}, {"source_db"=>"swh-deposit", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-deposit"}, {"source_db"=>"swh-indexer", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-indexer"}, {"source_db"=>"swh-web", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-web"}, {"source_db"=>"swh-mirror", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"swh-mirror"}, {"source_db"=>"swh-scrubber", "host"=>"localhost", "auth_user"=>"postgres", "port"=>5433, "alias"=>"staging-swh-scrubber"}]
*******************************************
+ Postgresql::Server::Database[swh-scrubber] =>
   parameters =>
      "connect_settings": {}
      "dbname": "swh-scrubber"
      "encoding": "UTF8"
      "istemplate": false
      "locale": "C.UTF-8"
      "owner": "swh-scrubber"
      "template": "template0"
*******************************************
+ Postgresql::Server::Database_grant[GRANT swh-scrubber - ALL - swh-scrubber] =>
   parameters =>
      "db": "swh-scrubber"
      "privilege": "ALL"
      "role": "swh-scrubber"
*******************************************
+ Postgresql::Server::Database_grant[swh-scrubber] =>
   parameters =>
      "db": "swh-scrubber"
      "privilege": "connect"
      "role": "guest"
*******************************************
+ Postgresql::Server::Db[swh-scrubber] =>
   parameters =>
      "dbname": "swh-scrubber"
      "encoding": "UTF8"
      "grant": "ALL"
      "istemplate": false
      "locale": "C.UTF-8"
      "owner": "swh-scrubber"
      "password": "swh-deploy-scrubber-db-password"
      "template": "template0"
      "user": "swh-scrubber"
*******************************************
+ Postgresql::Server::Grant[database:GRANT swh-scrubber - ALL - swh-scrubber] =>
   parameters =>
      "connect_settings": {}
      "db": "swh-scrubber"
      "ensure": "present"
      "group": "postgres"
      "object_arguments": []
      "object_name": "swh-scrubber"
      "object_type": "DATABASE"
      "onlyif_exists": false
      "port": 5433
      "privilege": "ALL"
      "psql_db": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "role": "swh-scrubber"
*******************************************
+ Postgresql::Server::Grant[database:swh-scrubber] =>
   parameters =>
      "connect_settings": {}
      "db": "swh-scrubber"
      "ensure": "present"
      "group": "postgres"
      "object_arguments": []
      "object_name": "swh-scrubber"
      "object_type": "DATABASE"
      "onlyif_exists": false
      "port": 5433
      "privilege": "connect"
      "psql_db": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "role": "guest"
*******************************************
+ Postgresql::Server::Role[swh-scrubber] =>
   parameters =>
      "connect_settings": {}
      "connection_limit": "-1"
      "createdb": false
      "createrole": false
      "db": "postgres"
      "ensure": "present"
      "inherit": true
      "login": true
      "module_workdir": "/tmp"
      "password_hash": "swh-deploy-scrubber-db-password"
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "replication": false
      "superuser": false
      "update_password": true
      "username": "swh-scrubber"
*******************************************
+ Postgresql_psql[ALTER DATABASE "swh-scrubber" OWNER TO "swh-scrubber"] =>
   parameters =>
      "connect_settings": {}
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_database JOIN pg_roles rol ON datdba = rol.oid WHERE datname = 'swh-scrubber' AND rolname = 'swh-scrubber'"
*******************************************
+ Postgresql_psql[ALTER ROLE "swh-scrubber" CONNECTION LIMIT -1] =>
   parameters =>
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_roles WHERE rolname = 'swh-scrubber' AND rolconnlimit = -1"
*******************************************
+ Postgresql_psql[ALTER ROLE "swh-scrubber" INHERIT] =>
   parameters =>
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_roles WHERE rolname = 'swh-scrubber' AND rolinherit = true"
*******************************************
+ Postgresql_psql[ALTER ROLE "swh-scrubber" LOGIN] =>
   parameters =>
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_roles WHERE rolname = 'swh-scrubber' AND rolcanlogin = true"
*******************************************
+ Postgresql_psql[ALTER ROLE "swh-scrubber" NOCREATEDB] =>
   parameters =>
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_roles WHERE rolname = 'swh-scrubber' AND rolcreatedb = false"
*******************************************
+ Postgresql_psql[ALTER ROLE "swh-scrubber" NOCREATEROLE] =>
   parameters =>
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_roles WHERE rolname = 'swh-scrubber' AND rolcreaterole = false"
*******************************************
+ Postgresql_psql[ALTER ROLE "swh-scrubber" NOREPLICATION] =>
   parameters =>
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_roles WHERE rolname = 'swh-scrubber' AND rolreplication = false"
*******************************************
+ Postgresql_psql[ALTER ROLE "swh-scrubber" NOSUPERUSER] =>
   parameters =>
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_roles WHERE rolname = 'swh-scrubber' AND rolsuper = false"
*******************************************
+ Postgresql_psql[ALTER ROLE swh-scrubber ENCRYPTED PASSWORD ****] =>
   parameters =>
      "command": "ALTER ROLE \"swh-scrubber\" ENCRYPTED PASSWORD 'swh-deploy-scrubber-db-password'"
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "sensitive": true
      "unless": "SELECT 1 FROM pg_shadow WHERE usename = 'swh-scrubber' AND passwd = 'md50d9c2eff14acb57e23860924ecbe3628'"
*******************************************
+ Postgresql_psql[CREATE DATABASE "swh-scrubber"] =>
   parameters =>
      "command": "CREATE DATABASE \"swh-scrubber\" WITH TEMPLATE = \"template0\" ENCODING = 'UTF8' LC_COLLATE = 'C.UTF-8' LC_CTYPE = 'C.UTF-8' "
      "connect_settings": {}
      "db": "postgres"
      "notify": ["Postgresql_psql[REVOKE CONNECT ON DATABASE \"swh-scrubber\" FROM public]"]
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_database WHERE datname = 'swh-scrubber'"
*******************************************
+ Postgresql_psql[CREATE ROLE swh-scrubber ENCRYPTED PASSWORD ****] =>
   parameters =>
      "command": "CREATE ROLE \"swh-scrubber\" ENCRYPTED PASSWORD 'swh-deploy-scrubber-db-password' LOGIN NOCREATEROLE NOCREATEDB NOSUPERUSER  CONNECTION LIMIT -1"
      "connect_settings": {}
      "cwd": "/tmp"
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "sensitive": true
      "unless": "SELECT 1 FROM pg_roles WHERE rolname = 'swh-scrubber'"
*******************************************
+ Postgresql_psql[REVOKE CONNECT ON DATABASE "swh-scrubber" FROM public] =>
   parameters =>
      "connect_settings": {}
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "refreshonly": true
*******************************************
+ Postgresql_psql[UPDATE pg_database SET datistemplate = false WHERE datname = 'swh-scrubber'] =>
   parameters =>
      "connect_settings": {}
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 FROM pg_database WHERE datname = 'swh-scrubber' AND datistemplate = false"
*******************************************
+ Postgresql_psql[grant:database:GRANT swh-scrubber - ALL - swh-scrubber] =>
   parameters =>
      "command": "GRANT ALL ON DATABASE \"swh-scrubber\" TO \"swh-scrubber\""
      "connect_settings": {}
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 WHERE has_database_privilege('swh-scrubber', 'swh-scrubber', 'CREATE') = true"
*******************************************
+ Postgresql_psql[grant:database:swh-scrubber] =>
   parameters =>
      "command": "GRANT CONNECT ON DATABASE \"swh-scrubber\" TO \"guest\""
      "connect_settings": {}
      "db": "postgres"
      "port": 5433
      "psql_group": "postgres"
      "psql_path": "/usr/bin/psql"
      "psql_user": "postgres"
      "unless": "SELECT 1 WHERE has_database_privilege('guest', 'swh-scrubber', 'CONNECT') = true"
*******************************************
*** End octocatalog-diff on db1.internal.staging.swh.network

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Apr 4 2022, 4:19 PM
Herald added a reviewer: System administrators. · View Herald TranscriptApr 4 2022, 4:19 PM
ardumont requested review of this revision.Apr 4 2022, 4:19 PM
Harbormaster completed remote builds in B28097: Diff 27189.Apr 4 2022, 4:19 PM
ardumont updated this revision to Diff 27198.Apr 4 2022, 4:53 PM

Update

Harbormaster completed remote builds in B28101: Diff 27198.Apr 4 2022, 4:53 PM
ardumont edited the test plan for this revision. (Show Details)Apr 4 2022, 4:54 PM
ardumont retitled this revision from wip: staging: Install new scrubber db in db1.staging to staging: Install new scrubber db in db1.staging.
vsellier accepted this revision.Apr 4 2022, 5:54 PM
This revision is now accepted and ready to land.Apr 4 2022, 5:54 PM
Closed by commit rSPSITEd0f7e9b9bf88: staging: Install new scrubber db in db1.staging (authored by ardumont). · Explain WhyApr 5 2022, 10:00 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rSPSITEd0f7e9b9bf88: staging: Install new scrubber db in db1.staging.
ardumont mentioned this in T4102: staging: Deploy swh-scrubber database and checkers.Apr 5 2022, 10:07 AM

Revision Contents
Changeset List

PathSize
data/
deployments/
staging/
12 lines

Diff 27209

View Options

data/deployments/staging/common.yaml

Loading...
About · Developer information · Legal