This is the version that brings extrinsic-metadata indexing
- tag swh-indexer v2.2.0
- T4406#88618: unstuck failing debian (buster) build [1]
- D8161: puppet: prepare new manfest for each indexer as journal client service [2]
- Stop puppet agent on indexer workers
- D8164, rSPSITE426cb09: Normalize service names consistently with cli and services
- rSPSITEda32d3: Further configuration fixes so all services starts
- Monitor journal client graph: https://grafana.softwareheritage.org/goto/YI27hIzVz?orgId=1
staging:
- run SQL migrations (to create origin_extrinsic_metadata table and related functions) [3]
- update and restart idx-storage server
- update and restart indexer journal client
- make sure the indexer journal client subscribed to the raw_extrinsic_metadata topic and is consuming from the start [5]
production:
- run SQL migrations (to create origin_extrinsic_metadata table and related functions) [4]
- update and restart idx-storage server
- update and restart indexer journal client
- make sure the indexer journal client subscribed to the raw_extrinsic_metadata topic and is consuming from the start [5]
[2] Existing deployed origin intrinsic metadata journal had a very large memory
consumption when it ran. So instead of letting that service run all index computations,
separate it into multiple ones dedicated to each index to compute so each can be
parametrized differently if need be.
[3]
swhstorage@storage1:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml version indexer --module-config-key=indexer_storage module: indexer current code version: 135 version: 134 swhstorage@storage1:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml upgrade indexer --to-version=135 --module-config-key=indexer_storage INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/135.sql' Migration to version 135 done swhstorage@storage1:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml version indexer --module-config-key=indexer_storage module: indexer current code version: 135 version: 135
[4]
swhstorage@saam:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml version indexer --module-config-key indexer_storage module: indexer current code version: 135 version: 134 swhstorage@saam:~$ swh db --config-file indexer.yml upgrade indexer --module-config-key=indexer_storage INFO:swh.core.db.db_utils:Executing migration script '/usr/lib/python3/dist-packages/swh/indexer/sql/upgrades/135.sql' Migration to version 135 done swhstorage@saam:~$ swh db --config-file /etc/softwareheritage/storage/indexer.yml version indexer --module-config-key indexer_storage module: indexer current code version: 135 version: 135
[5] new client with a new group id so it starts topics from the beginning