The swh-indexer stack is deployed on staging and the initial loading is done.
The volumes are quite low :
- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Nov 27 2020
Nov 27 2020
vsellier closed T2816: Enable the journal-writer for the swh-idx-storage in staging, a subtask of T2590: Finish the indexer -> swh-search pipeline, as Resolved.
vsellier committed rSPSITE2e1a65a3e33b: staging: Fix object storage configuration for indexers (authored by vsellier).
staging: Fix object storage configuration for indexers
vsellier committed rSPSITEa2a84c2efb3e: staging: configure idx-storage to write to kafka (authored by vsellier).
staging: configure idx-storage to write to kafka
this a description of the pipeline to clarify the interaction between the components (source: P883) :
Nov 26 2020
Nov 26 2020
vsellier changed the status of T2817: Enable the swh-search environment in staging, a subtask of T2590: Finish the indexer -> swh-search pipeline, from Open to Work in Progress.
vsellier renamed T2817: Enable the swh-search environment in staging from Enable the swh-search in staging to Enable the swh-search environment in staging.
T2814 needs to be released before
vsellier triaged T2816: Enable the journal-writer for the swh-idx-storage in staging as Normal priority.
Remove deprecation warning
vsellier committed rSPREdf717a7dd2fa: Reflect manual changes applied on journal0 (authored by vsellier).
Reflect manual changes applied on journal0
vsellier committed rDCIDXd92c241980db: swh.indexer.cli.journal_client: ensure the minimal configuration exists (authored by vsellier).
swh.indexer.cli.journal_client: ensure the minimal configuration exists
Improve test coverage and change mandatory configuration validation
Fix tests
the backfilling is complete (except for the metadatas). We will focus now on some clients to ensure all the local configuration is correct (T2814 for example), and then we will focus on exposing kafka to the outside.
The diff fixes the configuration issue but it seems there is another problem with the visits :
LGTM tested on staging
Nov 24 2020
Nov 24 2020
vsellier committed rSPSITE473c2b76b3f1: Configure backports to be able to install zfs packages (authored by vsellier).
Configure backports to be able to install zfs packages
The backfillng is done for several objects type and still in progress for revision, content, directory :
root@journal0:/opt/kafka/bin# for topic in $(./kafka-topics.sh --bootstrap-server $SERVER --list); do echo -n "$topic : "; ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list $SERVER --topic $topic | awk -F: '{s+=$3}END{print s}'; done
__consumer_offsets : 0
swh.journal.objects.content : 927440
swh.journal.objects.directory : 213279
swh.journal.objects.metadata_authority : 0
swh.journal.objects.metadata_fetcher : 0
swh.journal.objects.origin : 62892
swh.journal.objects.origin_visit : 68368
swh.journal.objects.origin_visit_status : 136721
swh.journal.objects.raw_extrinsic_metadata : 0
swh.journal.objects.release : 3101
swh.journal.objects.revision : 155746
swh.journal.objects.skipped_content : 189
swh.journal.objects.snapshot : 36046
swh.journal.objects_privileged.release : 0
swh.journal.objects_privileged.revision : 0I have some doubts on how to import the following object types and if they need to :
- swh.journal.objects.metadata_authority
- swh.journal.objects.metadata_fetcher
- swh.journal.objects.raw_extrinsic_metadata
- swh.journal.objects_privileged.release
- swh.journal.objects_privileged.revision
The topics were created with 64 partitions and a replication factor of 1 :
- the vm memory was increased from 12G to 20G (completely "pifometrique" approximated value)
- a new data disk of 500Go is attached to the VM (there are currently 300G of objects on storage1.staging)
- the kafka's logdir was configured to be stored on a zfs volume composed of only the new data disk :
root@journal0:~# apt install zfs-dkms ## reboot
Kafka is up and running on journal0.
The next steps are:
- tune the server as there is not a lot of disk space (and memory but only if needed) :
root@journal0:~# df -h Filesystem Size Used Avail Use% Mounted on udev 5.9G 0 5.9G 0% /dev tmpfs 1.2G 560K 1.2G 1% /run /dev/vda1 32G 7.2G 24G 24% / tmpfs 5.9G 8.0K 5.9G 1% /dev/shm tmpfs 5.0M 0 5.0M 0% /run/lock tmpfs 4.0M 0 4.0M 0% /sys/fs/cgroup tmpfs 244M 0 244M 0% /run/user/1025
root@journal0:~# free -h
total used free shared buff/cache available
Mem: 11Gi 6.5Gi 354Mi 11Mi 4.8Gi 4.9Gi
Swap: 0B 0B 0B- Create the topics as explained in T2520#48682 (with a smaller number of partition and a replication factor to 1 as we only have one staging server)
- Launch the backfill to populate kafka with the current content of the staging archive
vsellier committed rSPSITE13c87f140600: Declare the certificate for the journal0.staging's kafka (authored by vsellier).
Declare the certificate for the journal0.staging's kafka
vsellier committed rSPSITE9074d87bd6dd: Configure the kafka server for staging (authored by vsellier).
Configure the kafka server for staging
vsellier committed rSPSITEbf96179df42d: Ensure java is installed before trying to manipulate the java keystore (authored by vsellier).
Ensure java is installed before trying to manipulate the java keystore
vsellier committed rSPSITEedf036f29835: Move pgbouncer password alias to the main configuration (authored by vsellier).
Move pgbouncer password alias to the main configuration
vsellier committed rSPSITE7fa33c70aa99: Only store the sentry tokens in the credentials (authored by vsellier).
Only store the sentry tokens in the credentials
Add kafka::broker::password value
vsellier committed rSPPRIVC481fe7d73db8: Move pgbouncer alias to the main configuration (authored by vsellier).
Move pgbouncer alias to the main configuration
vsellier committed rSPPRIVC9cce52adf11e: Only store the sentry tokens in the credentials (authored by vsellier).
Only store the sentry tokens in the credentials
precision: the tests were only done on the internal plaintext listener without sasl authentication
Rename the zookeper cluster to match the kafka cluster name
vsellier committed rSENVf6eaeed656ef: vagrant: Increase memory to be able to run the services (authored by vsellier).
vagrant: Increase memory to be able to run the services
The configuration seems to work pretty well in vagrant :
root@journal0:/opt/kafka/bin# ./kafka-topics.sh --bootstrap-server 10.168.130.70:9092 --list | xargs -n1 -i{} ./kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.168.130.70:9092 --topic {} | grep -v consumer
swh.journal.objects.content:0:898
swh.journal.objects.directory:0:1172
swh.journal.objects.origin:0:26
swh.journal.objects.origin_visit:0:55
swh.journal.objects.origin_visit_status:0:2
swh.journal.objects.release:0:0
swh.journal.objects.revision:0:209
swh.journal.objects.snapshot:0:1I let the worker create the topics so there is only one partition. It will need to be adapted for the staging environment
Fix wrong indentation on the storage service configuration
Nov 23 2020
Nov 23 2020
Move the sentry dsn declarations with their friends
closed by rSPSITEed253c86d4f6
rebase
vsellier committed rSPSITEed253c86d4f6: Manage the parent directories of the kafka logdirs (authored by vsellier).
Manage the parent directories of the kafka logdirs
vagrant: fix es nodes fqdn
vsellier committed rSPSITEf58828fd5f86: vagrant: Fix network range allowed to connect to postgresql (authored by vsellier).
vagrant: Fix network range allowed to connect to postgresql
Avoid the declaration of the parent directories by using mkdir -p
Add staging-storage1 server
vsellier committed rSENV6710a971ac87: Fix staging-worker0 hostname to match the declaration in site.pp (authored by vsellier).
Fix staging-worker0 hostname to match the declaration in site.pp
vsellier committed rSENV0ef211987c01: Add journal0 certificates (used by kafka) (authored by vsellier).
Add journal0 certificates (used by kafka)
Nov 20 2020
Nov 20 2020
vagrant: add staging-storage0 node
Kafka needs a jre to run
All the loader where restarted on all the workers :
sudo clush -b -w @swh-workers 'apt-get update && apt-get -y upgrade -V' sudo clush -b -w @swh-workers 'puppet agent --enable && puppet agent --test' sudo clush -b -w @swh-workers 'systemctl default'
Automatic tasks restarted on worker01, the logs are under watch.
after upgrading the packages on worker01, the npm load was successful :
swhworker@worker01:~$ time SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_npm.yml swh loader run npm https://www.npmjs.com/package/bootstrap-vue
{'status': 'eventful', 'snapshot_id': '30d32aff7fab1a2c364dc5c61503b0aec3f9fb11'}The problem is not reproduced in staging but the worker and storage have the same package versions:
vsellier@worker0 ~ % apt list --upgradable Listing... Done python3-swh.deposit.client/unknown 0.6.0-1~swh1~bpo10+1 all [upgradable from: 0.5.0-1~swh1~bpo10+1] python3-swh.deposit.loader/unknown 0.6.0-1~swh1~bpo10+1 all [upgradable from: 0.5.0-1~swh1~bpo10+1] python3-swh.deposit/unknown 0.6.0-1~swh1~bpo10+1 all [upgradable from: 0.5.0-1~swh1~bpo10+1] python3-swh.indexer.storage/unknown 0.5.0-2~swh1~bpo10+1 all [upgradable from: 0.4.2-1~swh1~bpo10+1] python3-swh.indexer/unknown 0.5.0-2~swh1~bpo10+1 all [upgradable from: 0.4.2-1~swh1~bpo10+1] python3-swh.journal/unknown 0.5.1-1~swh1~bpo10+1 all [upgradable from: 0.5.0-1~swh1~bpo10+1] python3-swh.loader.git/unknown 0.5.0-1~swh1~bpo10+1 all [upgradable from: 0.4.1-1~swh1~bpo10+1] python3-swh.model/unknown 0.9.0-1~swh1~bpo10+1 all [upgradable from: 0.7.3-1~swh1~bpo10+1] python3-swh.storage/unknown 0.17.2-1~swh1~bpo10+1 all [upgradable from: 0.17.0-1~swh1~bpo10+1] python3-swh.vault/unknown 0.3.3-1~swh1~bpo10+1 all [upgradable from: 0.3.1-1~swh1~bpo10+1]
vsellier@storage1 ~ % apt list --upgradable Listing... Done libpq5/buster-pgdg 13.1-1.pgdg100+1 amd64 [upgradable from: 13.0-1.pgdg100+1] postgresql-13/buster-pgdg 13.1-1.pgdg100+1 amd64 [upgradable from: 13.0-1.pgdg100+1] postgresql-client-13/buster-pgdg 13.1-1.pgdg100+1 amd64 [upgradable from: 13.0-1.pgdg100+1] postgresql-client-common/buster-pgdg 223.pgdg100+1 all [upgradable from: 220.pgdg100+1] postgresql-client/buster-pgdg 13+223.pgdg100+1 all [upgradable from: 13+220.pgdg100+1] postgresql-common/buster-pgdg 223.pgdg100+1 all [upgradable from: 220.pgdg100+1] postgresql/buster-pgdg 13+223.pgdg100+1 all [upgradable from: 13+220.pgdg100+1] python3-swh.indexer.storage/unknown 0.5.0-2~swh1~bpo10+1 all [upgradable from: 0.4.2-1~swh1~bpo10+1] python3-swh.indexer/unknown 0.5.0-2~swh1~bpo10+1 all [upgradable from: 0.4.2-1~swh1~bpo10+1] python3-swh.journal/unknown 0.5.1-1~swh1~bpo10+1 all [upgradable from: 0.5.0-1~swh1~bpo10+1] python3-swh.model/unknown 0.9.0-1~swh1~bpo10+1 all [upgradable from: 0.7.3-1~swh1~bpo10+1] python3-swh.storage/unknown 0.17.2-1~swh1~bpo10+1 all [upgradable from: 0.17.0-1~swh1~bpo10+1]
- after upgrading storage1.staging, the exact problem is also present
- After upgrading the worker, everything goes well.
- puppet applied on worker01
- task by tasks tests :
- mercurial :
swhworker@worker01:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_mercurial.yml swh loader run mercurial https://foss.heptapod.net/fluiddyn/fluidfft
INFO:swh.loader.mercurial.Bundle20Loader:Load origin 'https://foss.heptapod.net/fluiddyn/fluidfft' with type 'hg'
{'status': 'eventful'}
swhworker@worker01:~$ SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_mercurial.yml swh loader run mercurial https://hg.mozilla.org/projects/nss
INFO:swh.loader.mercurial.Bundle20Loader:Load origin 'https://hg.mozilla.org/projects/nss' with type 'hg'
WARNING:swh.loader.mercurial.Bundle20Loader:No matching revision for tag NSS_3_15_5_BETA2 (hg changeset: e5d3ec1d9a35f7cac554543d52775092de9f6a01). Skipping
WARNING:swh.loader.mercurial.Bundle20Loader:No matching revision for tag NSS_3_15_5_BETA2 (hg changeset: 0000000000000000000000000000000000000000). Skipping
WARNING:swh.loader.mercurial.Bundle20Loader:No matching revision for tag NSS_3_18_RTM (hg changeset: 0000000000000000000000000000000000000000). Skipping
WARNING:swh.loader.mercurial.Bundle20Loader:No matching revision for tag NSS_3_18_RTM (hg changeset: 0000000000000000000000000000000000000000). Skipping
WARNING:swh.loader.mercurial.Bundle20Loader:No matching revision for tag NSS_3_24_BETA3 (hg changeset: 0000000000000000000000000000000000000000). Skipping
{'status': 'eventful'}- svn
root@worker01:~# SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_svn.yml swh loader run svn svn://svn.appwork.org/utils
INFO:swh.loader.svn.SvnLoader:Load origin 'svn://svn.appwork.org/utils' with type 'svn'
INFO:swh.loader.svn.SvnLoader:Processing revisions [3428-3436] for {'swh-origin': 'svn://svn.appwork.org/utils', 'remote_url': 'svn://svn.appwork.org/utils', 'local_url': b'/tmp/swh.loader.svn.dojsubkd-890577/utils', 'uuid': b'21714237-3853-44ef-a1f0-ef8f03a7d1fe'}
{'status': 'eventful'}- npm:
ko : https://sentry.softwareheritage.org/share/issue/363ef9d218ac4817a992b7dc9bf283a6/
root@worker01:~# SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_npm.yml swh loader run npm https://www.npmjs.com/package/bootstrap-vue
WARNING:swh.storage.retry:Retry adding a batch
WARNING:swh.storage.retry:Retry adding a batch
WARNING:swh.storage.retry:Retry adding a batch
ERROR:swh.loader.package.loader:Failed loading branch releases/2.18.0 for https://www.npmjs.com/package/bootstrap-vue
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/tenacity/__init__.py", line 333, in call
result = fn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/swh/storage/retry.py", line 117, in raw_extrinsic_metadata_add
return self.storage.raw_extrinsic_metadata_add(metadata)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 181, in meth_
return self.post(meth._endpoint_path, post_data)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 278, in post
return self._decode_response(response)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 352, in _decode_response
self.raise_for_status(response)
File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in raise_for_status
super().raise_for_status(response)
File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 342, in raise_for_status
raise exception from None
swh.core.api.RemoteException: <RemoteException 500 TypeError: ["__init__() got an unexpected keyword argument 'id'"]>root@worker01:~# SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_npm.yml swh loader run npm https://www.npmjs.com/package/vue
ERROR:swh.loader.package.loader:Failed loading branch releases/0.8.6 for https://www.npmjs.com/package/vue
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 424, in load
res = self._load_revision(p_info, origin)
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 539, in _load_revision
dl_artifacts = self.download_package(p_info, tmpdir)
File "/usr/lib/python3/dist-packages/swh/loader/package/loader.py", line 277, in download_package
return [download(p_info.url, dest=tmpdir, filename=p_info.filename)]
File "/usr/lib/python3/dist-packages/swh/loader/package/utils.py", line 80, in download
raise ValueError("Fail to query '%s'. Reason: %s" % (url, response.status_code))ValueError: Fail to query 'https://registry.npmjs.org/vue/-/vue-0.8.6.tgz'. Reason: 404
WARNING: swh.storage.retry:Retry adding a batch
WARNING:swh.storage.retry:Retry adding a batch
WARNING:swh.storage.retry:Retry adding a batch
ERROR:swh.loader.package.loader:Failed loading branch releases/3.0.1 for https://www.npmjs.com/package/vue
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/tenacity/__init.py", line 333, in call
result = fn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/swh/storage/retry.py", line 117, in raw_extrinsic_metadata_add
return self.storage.raw_extrinsic_metadata_add(metadata)
File "/usr/lib/python3/dist-packages/swh/core/api/init.py", line 181, in meth_
return self.post(meth._endpoint_path, post_data)
File "/usr/lib/python3/dist-packages/swh/core/api/init.py", line 278, in post
return self._decode_response(response)
File "/usr/lib/python3/dist-packages/swh/core/api/init.py", line 352, in _decode_response
self.raise_for_status(response)
File "/usr/lib/python3/dist-packages/swh/storage/api/client.py", line 29, in raise_for_status
super().raise_for_status(response)
File "/usr/lib/python3/dist-packages/swh/core/api/init.py", line 342, in raise_for_status
raise exception from None
swh.core.api.RemoteException: <RemoteException 500 TypeError: ["init__() got an unexpected keyword argument 'id'"]>
after applying the D4359 change on saam, the load is ok :
root@worker01:/etc/softwareheritage# sudo -u swhworker SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_git.yml swh loader run git https://github.com/SoftwareHeritage/puppet-swh-site
INFO:swh.loader.git.BulkLoader:Load origin 'https://github.com/SoftwareHeritage/puppet-swh-site' with type 'git'
Enumerating objects: 537, done.
Counting objects: 100% (537/537), done.
Compressing objects: 100% (326/326), done.
Total 19066 (delta 260), reused 445 (delta 194), pack-reused 18529
INFO:swh.loader.git.BulkLoader:Listed 3 refs for repo https://github.com/SoftwareHeritage/puppet-swh-site
{'status': 'eventful'}vsellier committed rSPSITE966b1d2eabd9: Add the mandatory cloud storage profile to bare metal storage servers (authored by vsellier).
Add the mandatory cloud storage profile to bare metal storage servers
vsellier updated the diff for D4539: Add mandatory cloud storage configuration on bare metal storage servers.
update the commit message