Page MenuHomeSoftware Heritage

staging: Deploy swh.loader.mercurial v2.1.0
Closed, ResolvedPublic

Description

Here goes the plan:

  • storage1.staging: Upgrade swh.storage to v0.35 (mind the migration step)
  • Make the staging check end-to-end for mercurial loader fail (as in production) by duplicating extids (rough sql magic inside).
  • workers: Upgrade swh.model and swh.loader.mercurial, restart workers
  • The failing staging check end-to-end should go green again
  • Clean up: Drop the duplicated extids

Event Timeline

ardumont changed the task status from Open to Work in Progress.Jul 29 2021, 12:30 PM
ardumont triaged this task as Unbreak Now! priority.
ardumont created this task.
ardumont moved this task from Backlog to in-progress on the System administration board.

From the staging webapp, we identify a revision [1]

That revision is currently with hex id 1850a087c1b6cb026470c738b2c7f82c6fada06d, so
let's duplicate an extid targetting this. As it's currently unexpected with the current
loader mercurial revision, this will fail the check

$ psql service=admin-staging
12:55:05 swh@db1:5432=> \conninfo
You are connected to database "swh" as user "swh" on host "db1.internal.staging.swh.network" (address "192.168.130.11") at port "5432".
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)

12:53:48 swh@db1:5432=> select * from extid where target='\x1850a087c1b6cb026470c738b2c7f82c6fada06d' and target_type='revision';
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
| extid_type |                   extid                    | target_type |                   target                   | extid_version |
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
| hg-nodeid  | \x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbf | revision    | \x1850a087c1b6cb026470c738b2c7f82c6fada06d |             0 |
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
(1 row)

Time: 10.173 ms
12:53:50 swh@db1:5432=> begin; insert into extid(extid_type, extid, target_type, target, extid_version) values('hg-nodeid', '\x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbe', 'revision', '\x1850a087c1b6cb026470c738b2c7f82c6fada06d', -1);
BEGIN
Time: 9.332 ms
INSERT 0 1
Time: 10.379 ms
12:54:57 *swh@db1:5432=> commit;
COMMIT
Time: 29.260 ms
12:55:02 swh@db1:5432=> select * from extid where target='\x1850a087c1b6cb026470c738b2c7f82c6fada06d' and target_type='revision';
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
| extid_type |                   extid                    | target_type |                   target                   | extid_version |
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
| hg-nodeid  | \x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbe | revision    | \x1850a087c1b6cb026470c738b2c7f82c6fada06d |            -1 |
| hg-nodeid  | \x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbf | revision    | \x1850a087c1b6cb026470c738b2c7f82c6fada06d |             0 |
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
(2 rows)

Time: 9.677 ms

The check fails.
Upgrade worker's mercurial loader package and restart service.
Trigger back a check, it successully loads now and we have the right extid_version entry to 1.

13:14:25 swh@db1:5432=> select * from extid where target='\x1850a087c1b6cb026470c738b2c7f82c6fada06d' and target_type='revision' order by extid_version desc;
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
| extid_type |                   extid                    | target_type |                   target                   | extid_version |
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
| hg-nodeid  | \x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbf | revision    | \x1850a087c1b6cb026470c738b2c7f82c6fada06d |             1 |
| hg-nodeid  | \x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbf | revision    | \x1850a087c1b6cb026470c738b2c7f82c6fada06d |             0 |
| hg-nodeid  | \x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbe | revision    | \x1850a087c1b6cb026470c738b2c7f82c6fada06d |            -1 |
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
(3 rows)

Time: 10.631 ms

Let's clean up the mess.

13:17:06 swh@db1:5432=> begin;  delete from extid where target='\x1850a087c1b6cb026470c738b2c7f82c6fada06d' and target_type='revision' and extid='\x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbe';
BEGIN
Time: 9.673 ms
DELETE 1
Time: 8.213 ms
13:17:12 *swh@db1:5432=> select * from extid where target='\x1850a087c1b6cb026470c738b2c7f82c6fada06d' and target_type='revision' order by extid_version desc ;
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
| extid_type |                   extid                    | target_type |                   target                   | extid_version |
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
| hg-nodeid  | \x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbf | revision    | \x1850a087c1b6cb026470c738b2c7f82c6fada06d |             1 |
| hg-nodeid  | \x6cd8ebdbc1581ec0ea838f224d2d4837675c3fbf | revision    | \x1850a087c1b6cb026470c738b2c7f82c6fada06d |             0 |
+------------+--------------------------------------------+-------------+--------------------------------------------+---------------+
(2 rows)

Time: 9.166 ms
13:17:14 *swh@db1:5432=> commit;
COMMIT
Time: 26.656 ms

Ok, we are good now on staging.

[1] https://webapp.staging.swh.network/browse/origin/directory/?origin_url=https://foss.heptapod.net/mercurial/hgview