Page MenuHomeSoftware Heritage

Setup dedicated kafka cluster on new rocquencourt hardware
Closed, ResolvedPublic


We have received machines to set up a new (full view) kafka cluster in rocquencourt; The cluster needs to be set up.

Related Objects

Event Timeline

olasd changed the task status from Open to Work in Progress.Aug 17 2020, 2:21 PM
olasd triaged this task as High priority.
olasd created this task.

After doing the basic kafka config, I've done the following to create the topics:

for object_type in content skipped_content directory revision release snapshot origin origin_visit origin_visit_status raw_extrinsic_metadata metadata_fetcher metadata_authority; do
    kafka-topics --bootstrap-server --create --config cleanup.policy=compact --partitions 256 --replication-factor 2 --topic swh.journal.objects.$object_type

for object_type in revision release; do
    kafka-topics --bootstrap-server --create --config cleanup.policy=compact --partitions 256 --replication-factor 2 --topic swh.journal.objects_privileged.$object_type

The missing topic (raw_extrinsic_metadata) has been handled as part of the migration in T3019. Closing.