Page MenuHomeSoftware Heritage

Deploy swh-indexer v2.2 on staging then production
Closed, ResolvedPublic

Description

This is the version that brings extrinsic-metadata indexing

staging:

  • run SQL migrations (to create origin_extrinsic_metadata table and related functions) [3]
  • update and restart idx-storage server
  • update and restart indexer journal client
    • content_mimetype ok
    • content_fossology_license ok
    • origin_intrinsic_metadata ok
    • T4412, D8165: extrinsic_metadata ko (diff D8165 as a workaround)
  • make sure the indexer journal client subscribed to the raw_extrinsic_metadata topic and is consuming from the start [5]

production:

  • run SQL migrations (to create origin_extrinsic_metadata table and related functions) [4]
  • update and restart idx-storage server
  • update and restart indexer journal client
  • make sure the indexer journal client subscribed to the raw_extrinsic_metadata topic and is consuming from the start [5]

[1] https://jenkins.softwareheritage.org/job/debian/job/packages/job/DCIDX/job/gbp-buildpackage/177/console

[2] Existing deployed origin intrinsic metadata journal had a very large memory
consumption when it ran. So instead of letting that service run all index computations,
separate it into multiple ones dedicated to each index to compute so each can be
parametrized differently if need be.

[3]

swhstorage@storage1:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml version indexer --module-config-key=indexer_storage
module: indexer
current code version: 135
version: 134
swhstorage@storage1:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml upgrade indexer --to-version=135 --module-config-key=indexer_storage
INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/135.sql'
Migration to version 135 done
swhstorage@storage1:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml version indexer --module-config-key=indexer_storage
module: indexer
current code version: 135
version: 135

[4]

swhstorage@saam:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml version indexer --module-config-key indexer_storage
module: indexer
current code version: 135
version: 134
swhstorage@saam:~$ swh db --config-file indexer.yml upgrade indexer --module-config-key=indexer_storage
INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/135.sql'
Migration to version 135 done
swhstorage@saam:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml version indexer --module-config-key indexer_storage
module: indexer
current code version: 135
version: 135

[5] new client with a new group id so it starts topics from the beginning

Event Timeline

vlorentz triaged this task as Normal priority.Jul 21 2022, 1:59 PM
vlorentz created this task.
  • unstuck failing debian (buster) build [1]

pyld dependency in the past for buster build:

python3-pyld | 0.6.8-1       | oldstable       | all
python3-pyld | 2.0.3-1       | stable          | all

So backported and installed in our debian repository.
Rough plan of what i did:

debcheckout python3-pyld
cd python3-pyld
sed -i 's/debcompat-helper (= 13)/debcompat-helper (= 12)/' debian/control
debchange --force-distribution --newversion "2.0.3-3~swh1" --distribution stretch-swh --package python3-pyld
git add debian/
git commit -m "d/*: Bump new release"
gbp-buildpackage --git-builder=sbuild
swh-debian-upload $HOME/debian/build-area/python-pyld_2.0.3-3~swh1_amd64.changes

And then inside the chroot:

(buster-amd64-sbuild)root@yavin4:/build/swh-indexer-z4dWq0# apt update
Ign:1 copy:/build/swh-indexer-z4dWq0/resolver-dX0BjT/apt_archive ./ InRelease
Get:2 copy:/build/swh-indexer-z4dWq0/resolver-dX0BjT/apt_archive ./ Release [963 B]
Ign:3 copy:/build/swh-indexer-z4dWq0/resolver-dX0BjT/apt_archive ./ Release.gpg
Hit:4 http://deb.debian.org/debian buster InRelease
Get:5 http://deb.debian.org/debian buster-backports InRelease [51.4 kB]
Ign:6 https://debian.softwareheritage.org buster-swh InRelease
Get:7 https://debian.softwareheritage.org buster-swh Release [2330 B]
Ign:8 https://debian.softwareheritage.org buster-swh Release.gpg
Get:9 https://debian.softwareheritage.org buster-swh/main amd64 Packages [54.0 kB]
Fetched 109 kB in 0s (691 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done
1 package can be upgraded. Run 'apt list --upgradable' to see it.
N: Download is performed unsandboxed as root as file '/build/swh-indexer-z4dWq0/resolver-dX0BjT/apt_archive/./InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)
(buster-amd64-sbuild)root@yavin4:/build/swh-indexer-z4dWq0# apt list --upgradable
Listing... Done
python3-pyld/unknown 2.0.3-3~swh1 all [upgradable from: 0.6.8-1]
N: There is 1 additional version. Please use the '-a' switch to see it
(buster-amd64-sbuild)root@yavin4:/build/swh-indexer-z4dWq0# apt upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following NEW packages will be installed:
  python3-cachetools python3-frozendict python3-lxml
The following packages will be upgraded:
  python3-pyld
1 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 1242 kB of archives.
After this operation, 4545 kB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://deb.debian.org/debian buster/main amd64 python3-cachetools all 3.1.0-2 [11.6 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 python3-frozendict all 1.2-1 [4588 B]
Get:3 http://deb.debian.org/debian buster/main amd64 python3-lxml amd64 4.3.2-1+deb10u4 [1163 kB]
Get:4 https://debian.softwareheritage.org buster-swh/main amd64 python3-pyld all 2.0.3-3~swh1 [62.0 kB]
Fetched 1242 kB in 0s (30.9 MB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package python3-cachetools.
(Reading database ... 25483 files and directories currently installed.)
Preparing to unpack .../python3-cachetools_3.1.0-2_all.deb ...
Unpacking python3-cachetools (3.1.0-2) ...
Selecting previously unselected package python3-frozendict.
Preparing to unpack .../python3-frozendict_1.2-1_all.deb ...
Unpacking python3-frozendict (1.2-1) ...
Selecting previously unselected package python3-lxml:amd64.
Preparing to unpack .../python3-lxml_4.3.2-1+deb10u4_amd64.deb ...
Unpacking python3-lxml:amd64 (4.3.2-1+deb10u4) ...
Preparing to unpack .../python3-pyld_2.0.3-3~swh1_all.deb ...
Unpacking python3-pyld (2.0.3-3~swh1) over (0.6.8-1) ...
Setting up python3-cachetools (3.1.0-2) ...
Setting up python3-frozendict (1.2-1) ...
Setting up python3-lxml:amd64 (4.3.2-1+deb10u4) ...
Setting up python3-pyld (2.0.3-3~swh1) ...
(buster-amd64-sbuild)root@yavin4:/build/swh-indexer-z4dWq0# su -l tony -s /bin/bash
su: warning: cannot change directory to /home/tony: No such file or directory
(buster-amd64-sbuild)tony@yavin4:/build/swh-indexer-z4dWq0$ cd swh-indexer
swh-indexer-2.2.0/                              swh-indexer_2.2.0-1~swh1~bpo10+1.debian.tar.xz  swh-indexer_2.2.0-1~swh1~bpo10+1.dsc            swh-indexer_2.2.0.orig.tar.gz
(buster-amd64-sbuild)tony@yavin4:/build/swh-indexer-z4dWq0$ cd swh-indexer-2.2.0/
(buster-amd64-sbuild)tony@yavin4:/build/swh-indexer-z4dWq0/swh-indexer-2.2.0$ python3 -m pytest
============================================================================================================= test session starts =============================================================================================================
platform linux -- Python 3.7.3, pytest-6.2.5, py-1.7.0, pluggy-0.13.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/build/swh-indexer-z4dWq0/swh-indexer-2.2.0/.hypothesis/examples')
rootdir: /build/swh-indexer-z4dWq0/swh-indexer-2.2.0, configfile: pytest.ini
plugins: celery-4.4.2, postgresql-3.1.3, hypothesis-3.71.11, swh.core-2.13, mock-1.10.4, swh.journal-1.1.0
collected 318 items

swh/indexer/tests/test_cli.py ..................                                                                                                                                                                                        [  5%]
swh/indexer/tests/test_codemeta.py .........                                                                                                                                                                                            [  8%]
swh/indexer/tests/test_fossology_license.py .........                                                                                                                                                                                   [ 11%]
...
=========================================================================================== 309 passed, 9 skipped, 12 warnings in 170.22s (0:02:50) ===========================================================================================

It works!

So now we just have to trigger back the build and it should pass.

ardumont updated the task description. (Show Details)
ardumont changed the task status from Open to Work in Progress.Jul 29 2022, 9:09 AM
ardumont moved this task from Backlog to in-progress on the System administration board.
ardumont updated the task description. (Show Details)
ardumont renamed this task from Deploy swh-indexer v2.2.0 on production and staging to Deploy swh-indexer v2.2 on production and staging.Jul 29 2022, 11:39 AM
ardumont renamed this task from Deploy swh-indexer v2.2 on production and staging to Deploy swh-indexer v2.2 on staging then production.
ardumont updated the task description. (Show Details)
ardumont added a subtask: Restricted Maniphest Task.Jul 29 2022, 1:54 PM
ardumont claimed this task.
ardumont moved this task from deployed/landed/monitoring to done on the System administration board.