Page MenuHomeSoftware Heritage

staging/production - Kafka access for ENEA mirror
Closed, ResolvedPublic

Description

An access token is needed on staging to start the test of the mirror in staging

Event Timeline

vsellier changed the task status from Open to Work in Progress.Oct 5 2021, 3:28 PM
vsellier triaged this task as Normal priority.
vsellier created this task.
vsellier moved this task from Backlog to in-progress on the System administration board.
vsellier added a subscriber: douardda.

token for the prod will be needed after that as well, thanks

export username=swh-enea
export password=XXXXX

opt/kafka/bin/kafka-configs.sh \
    --zookeeper ${zookeeper_servers}/kafka/softwareheritage \
    --alter \
    --add-config "SCRAM-SHA-256=[iterations=8192,password=$password],SCRAM-SHA-512=[password=$password]" \
    --entity-type users \
    --entity-name $username

opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation READ

/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:$username --operation READ


/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation DESCRIBE

/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:$username --operation DESCRIBE

/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --list --resource-pattern-type PREFIXED --topic swh.journal.objects. --principal User:$username

Credentials create in stagingd:

ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW) 

ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
vsellier renamed this task from staging - Kafka access for ENEA mirror to staging/production - Kafka access for ENEA mirror.Oct 5 2021, 4:24 PM

Production credentials created:

+ export zookeeper_servers=kafka1.internal.softwareheritage.org:2181
+ zookeeper_servers=kafka1.internal.softwareheritage.org:2181
+ export bootstrap_servers=kafka1.internal.softwareheritage.org:9092
+ bootstrap_servers=kafka1.internal.softwareheritage.org:9092
+ '[' -z swh-enea -o -z redacted ']'
+ set -eu
+ /opt/kafka/bin/kafka-configs.sh --zookeeper kafka1.internal.softwareheritage.org:2181/kafka/softwareheritage --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=redacted],SCRAM-SHA-512=[password=redacted]' --entity-type users --entity-name swh-enea
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'swh-enea'.
+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:swh-enea --operation READ
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-vse, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-olasd, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-olasd, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vse, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-content-replayer-s3, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-content-replayer-s3, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-seirl, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-seirl, host=*, operation=READ, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:swh-enea --operation READ
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-vlorentz, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=READ, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:swh-enea --operation DESCRIBE
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vse, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-olasd, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-olasd, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vse, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-content-replayer-s3, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-content-replayer-s3, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-seirl, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-seirl, host=*, operation=READ, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:swh-enea --operation DESCRIBE
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --list --resource-pattern-type PREFIXED --topic swh.journal.objects. --principal User:swh-enea
ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --list --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --principal User:swh-enea
ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
vsellier moved this task from in-progress to done on the System administration board.

credentials added on the credential database under the refs:

  • operations/kafka/credentials/staging/swh-enea
  • operations/kafka/credentials/production/swh-enea

The permissions were missing for consumer groups, so no consumer could get started at all.

I've used the opportunity to rename the credentials to something more descriptive: enea-stg-mirror-01 and enea-prod-mirror-01.

Thanks for having fixed the problem.
The missing command was something like:

# Allow READ on consumer groups prefixed with `$username-`
journal0 ~ % /opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --group ${username}- --allow-principal User:$username --operation READ