Page MenuHomeSoftware Heritage

Deploy indexer v2.0
Closed, MigratedEdits Locked

Description

Schema migration inside.

Plan (staging):

  • Backup tables to be dropped
  • idx.storage: Install new swh.indexer.storage version
  • idx.storage: Upgrade schema version
  • Upgrade worker versions
  • Restart indexer service
  • Checks [1]

Plan (production):

  • Backup tables to be dropped
  • idx.storage: Install new swh.indexer.storage version
  • idx.storage: Upgrade schema version
  • Upgrade worker versions
  • Restart indexer service
  • T4319#86682: Checks

[1]

10:04:11 swh-indexer@db1:5432=> select now(), count(*) from directory_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 08:04:52.166828+00 |     2 |
+-------------------------------+-------+
(1 row)

Time: 5.200 ms

Event Timeline

ardumont triaged this task as Normal priority.Jun 9 2022, 2:51 PM
ardumont created this task.
  • Backup tables that will get dropped [1]
  • current deployed db version: 133 [2]
  • current version to deploy: 134
  • Upgrade db version [3]

[1] Basic backup

09:32:15 swh-indexer@db1:5432=> create table origin_intrinsic_metadata_backup as table origin_intrinsic_metadata;
jjSELECT 1200895
Time: 276588.468 ms (04:36.588)
swh-indexer=# create table revision_intrinsic_metadata_backup as table revision_intrinsic_metadata;
SELECT 1345910

[2]

swhstorage@storage1:~$ swh db --config-file indexer.yml version indexer --module-config-key=indexer_storage
WARNING the database does not have a dbmodule table.
module: indexer
current code version: 134
version: 133

[3]

swhstorage@storage1:~$ swh db --config-file indexer.yml upgrade indexer --to-version=134 --module-config-key=indexer_storage
Warning: the database does not have a dbmodule table.
Write the module information (indexer) in the database? [Y/n]: y
INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/134.sql'
Migration to version 134 done
swhstorage@storage1:~$ swh db --config-file indexer.yml version indexer --module-config-key=indexer_storage
module: indexer
current code version: 134
version: 134
ardumont changed the task status from Open to Work in Progress.Jun 10 2022, 10:06 AM
ardumont updated the task description. (Show Details)
ardumont moved this task from Backlog to in-progress on the System administration board.

Ok, it's missing the indexes for the renewed origin_intrinsic_metadata in the upgrade script.

With those applied ^, workers are happier now.

10:23:09 swh-indexer@db1:5432=> select now(), count(*) from origin_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 08:23:15.108839+00 |    10 |
+-------------------------------+-------+
(1 row)

Time: 5.906 ms
10:23:15 swh-indexer@db1:5432=> select now(), count(*) from directory_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 08:23:19.821297+00 |    37 |
+-------------------------------+-------+
(1 row)

Time: 5.276 ms
  • Backup:
softwareheritage-indexer=# create table origin_intrinsic_metadata_backup as table origin_intrinsic_metadata;
SELECT 22359694
softwareheritage-indexer=# create table revision_intrinsic_metadata_backup as table revision_intrinsic_metadata;
SELECT 16955557
softwareheritage-indexer=# alter table origin_intrinsic_metadata_backup owner to swhstorage;
ALTER TABLE
softwareheritage-indexer=# alter table revision_intrinsic_metadata_backup owner to swhstorage;
ALTER TABLE

Migrate schema:

swhstorage@saam:~$ swh db --config-file indexer.yml upgrade indexer --to-version=134 --module-config-key=indexer_storage
INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/134.sql'
Migration to version 134 done

I had to install python3-swh.indexer for the cli to accept to work though...
So something must be off in the indexer packaging possibly.

workers restarting.
At least one is done and stuff are being written accordingly.

17:32:42 softwareheritage-indexer@somerset:5432=> select now(), count(*) from directory_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 15:32:43.312589+00 |    20 |
+-------------------------------+-------+
(1 row)

Time: 17.302 ms
17:32:43 softwareheritage-indexer@somerset:5432=> select now(), count(*) from origin_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 15:32:47.687563+00 |    20 |
+-------------------------------+-------+
(1 row)

Time: 7.385 ms
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.