Page MenuHomeSoftware Heritage
Create Task
Maniphest T4319

Deploy indexer v2.0
Closed, MigratedEdits Locked
Actions

Description

Schema migration inside.

Plan (staging):

  • Backup tables to be dropped
  • idx.storage: Install new swh.indexer.storage version
  • idx.storage: Upgrade schema version
  • Upgrade worker versions
  • Restart indexer service
  • Checks [1]

Plan (production):

  • Backup tables to be dropped
  • idx.storage: Install new swh.indexer.storage version
  • idx.storage: Upgrade schema version
  • Upgrade worker versions
  • Restart indexer service
  • T4319#86682: Checks

[1]

10:04:11 swh-indexer@db1:5432=> select now(), count(*) from directory_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 08:04:52.166828+00 |     2 |
+-------------------------------+-------+
(1 row)

Time: 5.200 ms

Revisions and Commits

rDCIDX Metadata indexer
D7978rDCIDX710467138ac9 upgrades/134: Add missing index creation

Related Objects

Event Timeline

ardumont triaged this task as Normal priority.Jun 9 2022, 2:51 PM
ardumont created this task.
ardumont mentioned this in D7971: common/archive: Ensure backward compatibility with swh-indexer 1.x.Jun 9 2022, 6:38 PM
ardumont updated the task description. (Show Details)Jun 10 2022, 9:28 AM
ardumont updated the task description. (Show Details)Jun 10 2022, 10:04 AM
  • Backup tables that will get dropped [1]
  • current deployed db version: 133 [2]
  • current version to deploy: 134
  • Upgrade db version [3]

[1] Basic backup

09:32:15 swh-indexer@db1:5432=> create table origin_intrinsic_metadata_backup as table origin_intrinsic_metadata;
jjSELECT 1200895
Time: 276588.468 ms (04:36.588)
swh-indexer=# create table revision_intrinsic_metadata_backup as table revision_intrinsic_metadata;
SELECT 1345910

[2]

swhstorage@storage1:~$ swh db --config-file indexer.yml version indexer --module-config-key=indexer_storage
WARNING the database does not have a dbmodule table.
module: indexer
current code version: 134
version: 133

[3]

swhstorage@storage1:~$ swh db --config-file indexer.yml upgrade indexer --to-version=134 --module-config-key=indexer_storage
Warning: the database does not have a dbmodule table.
Write the module information (indexer) in the database? [Y/n]: y
INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/134.sql'
Migration to version 134 done
swhstorage@storage1:~$ swh db --config-file indexer.yml version indexer --module-config-key=indexer_storage
module: indexer
current code version: 134
version: 134
ardumont changed the task status from Open to Work in Progress.Jun 10 2022, 10:06 AM
ardumont updated the task description. (Show Details)
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont added a comment.Jun 10 2022, 10:14 AM

staging
...

  • Checks

Although [1] [2]

[1] https://sentry.softwareheritage.org/share/issue/c224d3ab1804452e84d68170ab55590e/

[2] https://sentry.softwareheritage.org/share/issue/6e2d70aa3c4e423fa763a456130a3cc9/

ardumont added a comment.Jun 10 2022, 10:17 AM
In T4319#86628, @ardumont wrote:

staging
...

  • Checks

Although [1] [2]

[1] https://sentry.softwareheritage.org/share/issue/c224d3ab1804452e84d68170ab55590e/

[2] https://sentry.softwareheritage.org/share/issue/6e2d70aa3c4e423fa763a456130a3cc9/

Ok, it's missing the indexes for the renewed origin_intrinsic_metadata in the upgrade script.

ardumont added a revision: D7978: upgrades/134: Add missing index creation.Jun 10 2022, 10:23 AM
ardumont added a comment.Jun 10 2022, 10:24 AM

With those applied ^, workers are happier now.

10:23:09 swh-indexer@db1:5432=> select now(), count(*) from origin_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 08:23:15.108839+00 |    10 |
+-------------------------------+-------+
(1 row)

Time: 5.906 ms
10:23:15 swh-indexer@db1:5432=> select now(), count(*) from directory_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 08:23:19.821297+00 |    37 |
+-------------------------------+-------+
(1 row)

Time: 5.276 ms
ardumont added a commit: rDCIDX710467138ac9: upgrades/134: Add missing index creation.Jun 10 2022, 10:35 AM
ardumont added a comment.Jun 10 2022, 5:20 PM
  • Backup:
softwareheritage-indexer=# create table origin_intrinsic_metadata_backup as table origin_intrinsic_metadata;
SELECT 22359694
softwareheritage-indexer=# create table revision_intrinsic_metadata_backup as table revision_intrinsic_metadata;
SELECT 16955557
softwareheritage-indexer=# alter table origin_intrinsic_metadata_backup owner to swhstorage;
ALTER TABLE
softwareheritage-indexer=# alter table revision_intrinsic_metadata_backup owner to swhstorage;
ALTER TABLE
ardumont updated the task description. (Show Details)Jun 10 2022, 5:21 PM
ardumont added a comment.Jun 10 2022, 5:23 PM

Migrate schema:

swhstorage@saam:~$ swh db --config-file indexer.yml upgrade indexer --to-version=134 --module-config-key=indexer_storage
INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/134.sql'
Migration to version 134 done

I had to install python3-swh.indexer for the cli to accept to work though...
So something must be off in the indexer packaging possibly.

ardumont updated the task description. (Show Details)Jun 10 2022, 5:27 PM
ardumont moved this task from in-progress to deployed/landed/monitoring on the System administration board.Jun 10 2022, 5:32 PM

workers restarting.
At least one is done and stuff are being written accordingly.

17:32:42 softwareheritage-indexer@somerset:5432=> select now(), count(*) from directory_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 15:32:43.312589+00 |    20 |
+-------------------------------+-------+
(1 row)

Time: 17.302 ms
17:32:43 softwareheritage-indexer@somerset:5432=> select now(), count(*) from origin_intrinsic_metadata;
+-------------------------------+-------+
|              now              | count |
+-------------------------------+-------+
| 2022-06-10 15:32:47.687563+00 |    20 |
+-------------------------------+-------+
(1 row)

Time: 7.385 ms
ardumont updated the task description. (Show Details)Jun 10 2022, 5:33 PM
ardumont closed this task as Resolved.Jun 13 2022, 1:11 PM
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:07 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal