Page MenuHomeSoftware Heritage

Find a way to properly open the kafka brokers to the internet
Started, Work in Progress, HighPublic

Description

The kafka brokers need to be accessible from the internet, so our mirrors can subscribe to the topics and process messages.

We need to figure out:

  • frontend/proxying
  • TLS
  • authentication
  • authorization

For reference:

There's a strong chance that the journal code will need to be adapted to allow passing the proper settings to the kafka libraries.

Event Timeline

olasd triaged this task as High priority.Jun 18 2019, 4:02 PM
olasd created this task.
olasd changed the task status from Open to Work in Progress.Aug 23 2019, 6:45 PM

A new Kafka cluster has been spun up on azure virtual machines, with 6 machines each with 8TB of storage available.

A Kafka Mirror Maker has been setup by hand on getty to pull the data from the cluster in Rocquencourt to the cluster on Azure (only on the content topic for now).

My working theory for now is to use the Rocquencourt cluster as a low-latency buffer in front of the Azure cluster.

The next step is to lock this cluster down with SASL authentication, and to give it public ip addresses and TLS setup so it can be opened to the internet.

olasd added a comment.Aug 26 2019, 8:57 AM

The content topic has fully replicated to the new cluster over the weekend.

I've now added replication for the other topics.

olasd added a comment.Feb 7 2020, 7:08 PM

Following documentation with the following links:

We've updated settings of the azure kafka cluster to:

  • enable TLS
  • enable SASL authentication
  • open the TLS port to the world

Latest commit in puppet: rSPSITE250ea3a376cc
Latest commit in terraform: rSPRE1acb92757799

With the current deployment, the inter-broker, internal communication still happens in plaintext; clients local to the swh infra can still access the cluster without authentication or encryption. However, an internal port has been opened for TLS communication (port 9094 on all brokers).

Adding a new user and credentials is done with the following command:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka01.euwest.azure.internal.softwareheritage.org:2181,kafka02.euwest.azure.internal.softwareheritage.org:2181,kafka03.euwest.azure.internal.softwareheritage.org:2181,kafka04.euwest.azure.internal.softwareheritage.org:2181,kafka05.euwest.azure.internal.softwareheritage.org:2181,kafka06.euwest.azure.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --add-config 'SCRAM-SHA-256=[iterations=8192,password=<password>],SCRAM-SHA-512=[password=<password>]' \
    --entity-type users \
    --entity-name <username>

Removing a user's credentials is done with the following:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka01.euwest.azure.internal.softwareheritage.org:2181,kafka02.euwest.azure.internal.softwareheritage.org:2181,kafka03.euwest.azure.internal.softwareheritage.org:2181,kafka04.euwest.azure.internal.softwareheritage.org:2181,kafka05.euwest.azure.internal.softwareheritage.org:2181,kafka06.euwest.azure.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --delete-config 'SCRAM-SHA-256,SCRAM-SHA-512' \
    --entity-type users \
    --entity-name <username>

(I'll have puppet generate wrapper scripts for this mess).

Running a client on my laptop at Inria pointed at the public IP addresses, using the following SASL config, works properly:

c = confluent_kafka.Consumer({
    'group.id': 'olasd-test-sasl-1',
    'bootstrap.servers': ','.join('kafka%02d.euwest.azure.softwareheritage.org:9093' % i for i in range(1,7)),
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'SCRAM-SHA-512',
    'sasl.username': '<username>',
    'sasl.password': '<password>',
    'debug': 'consumer',
})

Unfortunately, port 9093 is firewalled between our infra at SESI and the outside world. For now, the consumer on uffizi has been re-configured to use port 9094 (internal tls) and a standalone credential.

The next step will be introducing Authorization.

olasd added a comment.Feb 10 2020, 3:54 PM

Using swh.journal.client:

c = swh.journal.client.JournalClient(**{
    'group_id': 'olasd-test-sasl-1',
    'brokers': ['kafka%02d.euwest.azure.softwareheritage.org:9093' % i for i in range(1,7)],
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'SCRAM-SHA-512',
    'sasl.username': '<username>',
    'sasl.password': '<password>',
    'debug': 'consumer',
})

(yes, passing dotted config parameters in kwargs is... not the cleanest)