Page MenuHomeSoftware Heritage
Create Task
Maniphest T3633

staging/production - Kafka access for ENEA mirror
Closed, MigratedEdits Locked
Actions

Description

An access token is needed on staging to start the test of the mirror in staging

Event Timeline

vsellier changed the task status from Open to Work in Progress.Oct 5 2021, 3:28 PM
vsellier triaged this task as Normal priority.
vsellier created this task.
vsellier moved this task from Backlog to in-progress on the System administration board.
vsellier added a subscriber: douardda.
douardda added a comment.Oct 5 2021, 3:40 PM

token for the prod will be needed after that as well, thanks

vsellier added a comment.Oct 5 2021, 3:49 PM
export username=swh-enea
export password=XXXXX

opt/kafka/bin/kafka-configs.sh \
    --zookeeper ${zookeeper_servers}/kafka/softwareheritage \
    --alter \
    --add-config "SCRAM-SHA-256=[iterations=8192,password=$password],SCRAM-SHA-512=[password=$password]" \
    --entity-type users \
    --entity-name $username

opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation READ

/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:$username --operation READ


/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation DESCRIBE

/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:$username --operation DESCRIBE

/opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --list --resource-pattern-type PREFIXED --topic swh.journal.objects. --principal User:$username
vsellier added a comment.EditedOct 5 2021, 3:50 PM

Credentials create in stagingd:

ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW) 

ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
vsellier renamed this task from staging - Kafka access for ENEA mirror to staging/production - Kafka access for ENEA mirror.Oct 5 2021, 4:24 PM
vsellier added a comment.Oct 5 2021, 4:49 PM

Production credentials created:

+ export zookeeper_servers=kafka1.internal.softwareheritage.org:2181
+ zookeeper_servers=kafka1.internal.softwareheritage.org:2181
+ export bootstrap_servers=kafka1.internal.softwareheritage.org:9092
+ bootstrap_servers=kafka1.internal.softwareheritage.org:9092
+ '[' -z swh-enea -o -z redacted ']'
+ set -eu
+ /opt/kafka/bin/kafka-configs.sh --zookeeper kafka1.internal.softwareheritage.org:2181/kafka/softwareheritage --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=redacted],SCRAM-SHA-512=[password=redacted]' --entity-type users --entity-name swh-enea
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'swh-enea'.
+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:swh-enea --operation READ
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-vse, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-olasd, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-olasd, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vse, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-content-replayer-s3, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-content-replayer-s3, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-seirl, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-seirl, host=*, operation=READ, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:swh-enea --operation READ
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-vlorentz, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=READ, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:swh-enea --operation DESCRIBE
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vse, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-olasd, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-olasd, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vse, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-content-replayer-s3, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-content-replayer-s3, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-seirl, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-seirl, host=*, operation=READ, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --allow-principal User:swh-enea --operation DESCRIBE
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-vlorentz, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --list --resource-pattern-type PREFIXED --topic swh.journal.objects. --principal User:swh-enea
ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW) 

+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --list --resource-pattern-type PREFIXED --topic swh.journal.objects_privileged. --principal User:swh-enea
ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)
vsellier closed this task as Resolved.Oct 5 2021, 4:51 PM
vsellier moved this task from in-progress to done on the System administration board.

credentials added on the credential database under the refs:

  • operations/kafka/credentials/staging/swh-enea
  • operations/kafka/credentials/production/swh-enea
olasd added a subscriber: olasd.Oct 15 2021, 6:53 PM

The permissions were missing for consumer groups, so no consumer could get started at all.

I've used the opportunity to rename the credentials to something more descriptive: enea-stg-mirror-01 and enea-prod-mirror-01.

vsellier added a comment.Oct 15 2021, 7:32 PM

Thanks for having fixed the problem.
The missing command was something like:

# Allow READ on consumer groups prefixed with `$username-`
journal0 ~ % /opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --group ${username}- --allow-principal User:$username --operation READ
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:04 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal