Find a way to properly open the kafka brokers to the internet
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	olasd
	Jun 18 2019, 4:02 PM

Description

The kafka brokers need to be accessible from the internet, so our mirrors can subscribe to the topics and process messages.

We need to figure out:

frontend/proxying
TLS
authentication
authorization

For reference:

proxy-based deployment with TLS offloading : https://medium.com/code-tech/kafka-in-aws-with-ssl-offloading-using-load-balancer-c337da1435c3
Authentication: https://docs.confluent.io/current/kafka/authentication_sasl/index.html
Authorization: https://docs.confluent.io/current/kafka/authorization.html

There's a strong chance that the journal code will need to be adapted to allow passing the proper settings to the kafka libraries.

Related Objects

Mentioned In: T3188: staging/journal: create douardda credentials
T1017: Estimate for Kafka cluster specifications
Mentioned Here: rSPSITEe1a942059f00: Add superusers to the kafka cluster setup
rSPSITE9993a81ffc7a: Enable kafka ACLs
rSPSITE250ea3a376cc: Re-jig kafka listeners to have a properly advertised TLS listener on public IPs
rSPRE1acb92757799: Add public ip addresses to kafka brokers

Event Timeline

olasd triaged this task as High priority.Jun 18 2019, 4:02 PM

olasd created this task.

olasd mentioned this in T1017: Estimate for Kafka cluster specifications.Aug 23 2019, 6:42 PM

A new Kafka cluster has been spun up on azure virtual machines, with 6 machines each with 8TB of storage available.

A Kafka Mirror Maker has been setup by hand on getty to pull the data from the cluster in Rocquencourt to the cluster on Azure (only on the content topic for now).

My working theory for now is to use the Rocquencourt cluster as a low-latency buffer in front of the Azure cluster.

The next step is to lock this cluster down with SASL authentication, and to give it public ip addresses and TLS setup so it can be opened to the internet.

The content topic has fully replicated to the new cluster over the weekend.

I've now added replication for the other topics.

Following documentation with the following links:

We've updated settings of the azure kafka cluster to:

enable TLS
enable SASL authentication
open the TLS port to the world

Latest commit in puppet: rSPSITE250ea3a376cc
Latest commit in terraform: rSPRE1acb92757799

With the current deployment, the inter-broker, internal communication still happens in plaintext; clients local to the swh infra can still access the cluster without authentication or encryption. However, an internal port has been opened for TLS communication (port 9094 on all brokers).

Adding a new user and credentials is done with the following command:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka01.euwest.azure.internal.softwareheritage.org:2181,kafka02.euwest.azure.internal.softwareheritage.org:2181,kafka03.euwest.azure.internal.softwareheritage.org:2181,kafka04.euwest.azure.internal.softwareheritage.org:2181,kafka05.euwest.azure.internal.softwareheritage.org:2181,kafka06.euwest.azure.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --add-config 'SCRAM-SHA-256=[iterations=8192,password=<password>],SCRAM-SHA-512=[password=<password>]' \
    --entity-type users \
    --entity-name <username>

Removing a user's credentials is done with the following:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka01.euwest.azure.internal.softwareheritage.org:2181,kafka02.euwest.azure.internal.softwareheritage.org:2181,kafka03.euwest.azure.internal.softwareheritage.org:2181,kafka04.euwest.azure.internal.softwareheritage.org:2181,kafka05.euwest.azure.internal.softwareheritage.org:2181,kafka06.euwest.azure.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --delete-config 'SCRAM-SHA-256,SCRAM-SHA-512' \
    --entity-type users \
    --entity-name <username>

(I'll have puppet generate wrapper scripts for this mess).

Running a client on my laptop at Inria pointed at the public IP addresses, using the following SASL config, works properly:

c = confluent_kafka.Consumer({
    'group.id': 'olasd-test-sasl-1',
    'bootstrap.servers': ','.join('kafka%02d.euwest.azure.softwareheritage.org:9093' % i for i in range(1,7)),
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'SCRAM-SHA-512',
    'sasl.username': '<username>',
    'sasl.password': '<password>',
    'debug': 'consumer',
})

Unfortunately, port 9093 is firewalled between our infra at SESI and the outside world. For now, the consumer on uffizi has been re-configured to use port 9094 (internal tls) and a standalone credential.

The next step will be introducing Authorization.

Using swh.journal.client:

c = swh.journal.client.JournalClient(**{
    'group_id': 'olasd-test-sasl-1',
    'brokers': ['kafka%02d.euwest.azure.softwareheritage.org:9093' % i for i in range(1,7)],
    'security.protocol': 'SASL_SSL',
    'sasl.mechanisms': 'SCRAM-SHA-512',
    'sasl.username': '<username>',
    'sasl.password': '<password>',
    'debug': 'consumer',
})

(yes, passing dotted config parameters in kwargs is... not the cleanest)

The new cluster in rocquencourt is using the built-in Kafka ACLs now (9993a81ffc7a1c8bd519b33ae63ac1145105f624).

Connections to the internal plaintext port are mapped to a built-in ANONYMOUS user. To keep the status quo, this user has been configured as superuser (e1a942059f0081cfb1e69baeab9defb5d93d3776), until we've configured producers and consumers not to need that.

User management

Usernames follow the pattern : <namespace>-<username>. For instance: swh-olasd is a user for @olasd in the swh namespace.

User creation

To create a user, run the following command:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka1.internal.softwareheritage.org,kafka2.internal.softwareheritage.org,kafka3.internal.softwareheritage.org,kafka4.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --add-config 'SCRAM-SHA-256=[iterations=8192,password=<password>],SCRAM-SHA-512=[password=<password>]' \
    --entity-type users \
    --entity-name <full-username>

The password is stored, hashed, in zookeeper.

User deletion

To delete a user, run the following command:

/opt/kafka/bin/kafka-configs.sh \
    --zookeeper kafka1.internal.softwareheritage.org,kafka2.internal.softwareheritage.org,kafka3.internal.softwareheritage.org,kafka4.internal.softwareheritage.org:2181/kafka/softwareheritage \
    --alter \
    --delete-config 'SCRAM-SHA-256,SCRAM-SHA-512' \
    --entity-type users \
    --entity-name <full-username>

Superuser

To set a user as superuser (allowing it to create topics and perform other management tasks), create it with the previous command, and add it to the kafka configuration in puppet (swh-site/data/common/kafka.yaml). The cluster needs to be restarted for the config change to apply.

Kafka command configuration

To use the Kafka CLI utilities, the following settings need to be configured in a kafka.properties file, passed to the various command line utilities (either as --command-config or --consumer.config, depending on the command line utility):

sasl.mechanism=SCRAM-SHA-512
security.protocol=SASL_SSL
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required \
  username="<username>" \
  password="<password>";

ACLs

By default, users don't have any read or write access to any data on the cluster.

Consumer ACLs

We box consumers in a given consumer group prefix, to allow multiple consumers for a given user but avoiding users interfering with one another.
We only allow reads on topics starting with a given prefix.

bootstrap_servers=kafka1.internal.softwareheritage.org:9092,kafka2.internal.softwareheritage.org:9092,kafka3.internal.softwareheritage.org:9092,kafka4.internal.softwareheritage.org:9092
username=<full-username>

# Allow READ and DESCRIBE on unprivileged topics
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation READ
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation DESCRIBE

# Allow READ on consumer groups prefixed with `$username-`
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --group ${username}- --allow-principal User:$username --operation READ

To allow a consumer to read privileged (non-anonymized) topics:

bootstrap_servers=kafka1.internal.softwareheritage.org:9092,kafka2.internal.softwareheritage.org:9092,kafka3.internal.softwareheritage.org:9092,kafka4.internal.softwareheritage.org:9092
username=<full-username>

# Allow READ and DESCRIBE on **privileged** topics
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:$username --operation READ
/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:$username --operation DESCRIBE

Producer ACLs

TBD.

vsellier mentioned this in T3188: staging/journal: create douardda credentials.Mar 30 2021, 12:18 PM

This task has been migrated to GitLab.

Find a way to properly open the kafka brokers to the internetClosed, MigratedEdits LockedActions