Page MenuHomeSoftware Heritage

Configure the kafka server for staging
ClosedPublic

Authored by vsellier on Nov 23 2020, 4:07 PM.

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vsellier created this revision.

Fix wrong indentation on the storage service configuration

The journal configuration was not at the right place

The configuration seems to work pretty well in vagrant :

root@journal0:/opt/kafka/bin# ./kafka-topics.sh --bootstrap-server 10.168.130.70:9092 --list | xargs -n1 -i{} ./kafka-run-class.sh kafka.tools.GetOffsetShell  --broker-list 10.168.130.70:9092 --topic {} | grep -v consumer
swh.journal.objects.content:0:898
swh.journal.objects.directory:0:1172
swh.journal.objects.origin:0:26
swh.journal.objects.origin_visit:0:55
swh.journal.objects.origin_visit_status:0:2
swh.journal.objects.release:0:0
swh.journal.objects.revision:0:209
swh.journal.objects.snapshot:0:1

I let the worker create the topics so there is only one partition. It will need to be adapted for the staging environment

This revision is now accepted and ready to land.Nov 24 2020, 9:56 AM

Rename the zookeper cluster to match the kafka cluster name

precision: the tests were only done on the internal plaintext listener without sasl authentication