Page MenuHomeSoftware Heritage

Setup dedicated kafka cluster on new rocquencourt hardware
Started, Work in Progress, HighPublic

Description

We have received machines to set up a new (full view) kafka cluster in rocquencourt; The cluster needs to be set up.

Event Timeline

olasd changed the task status from Open to Work in Progress.Aug 17 2020, 2:21 PM
olasd triaged this task as High priority.
olasd created this task.

After doing the basic kafka config, I've done the following to create the topics:

for object_type in content skipped_content directory revision release snapshot origin origin_visit origin_visit_status raw_extrinsic_metadata metadata_fetcher metadata_authority; do
    kafka-topics --bootstrap-server kafka1.internal.softwareheritage.org:9092 --create --config cleanup.policy=compact --partitions 256 --replication-factor 2 --topic swh.journal.objects.$object_type
done

for object_type in revision release; do
    kafka-topics --bootstrap-server kafka1.internal.softwareheritage.org:9092 --create --config cleanup.policy=compact --partitions 256 --replication-factor 2 --topic swh.journal.objects_privileged.$object_type
done