Page MenuHomeSoftware Heritage
Create Task
Maniphest T4265

Create test topics on staging's kafka for the mirror tests
Closed, MigratedEdits Locked
Actions

Description

  • Create the usual swh topics under the swh.test.journal... namespace on the staging's kafka with 16 partitions.
  • Provide 2 users:
    • mirror-test-rw for dataset intialisation
    • mirror-test-ro to run the tests
  • Provide a set a command lines to manage the topics (reset/recreate/....)

Revisions and Commits

rSPSITE puppet-swh-site
D7962rSPSITE395c4a2a79de kafka: add more options to the user management script

Related Objects

Event Timeline

vsellier triaged this task as Normal priority.May 23 2022, 1:48 PM
vsellier created this task.
douardda added a comment.May 23 2022, 3:26 PM

topics to be created under the `swh.test' prefix (or similar).

vsellier added a comment.May 23 2022, 3:27 PM

thanks, I forgot to mention that

vsellier updated the task description. (Show Details)May 23 2022, 3:27 PM
vsellier changed the task status from Open to Work in Progress.Jun 2 2022, 10:19 AM
vsellier added a comment.Jun 3 2022, 12:16 PM

Topics created:

storage1 /opt/kafka/bin% TOPICS="content directory extid metadata_authority metadata_fetcher origin origin_visit origin_visit_status raw_extrinsic_metadata release revision skipped_content snapshot"

storage1 /opt/kafka/bin% for object_type in $(echo ${TOPICS}); do
    ./kafka-topics.sh --bootstrap-server $(hostname -f):9092 --create \
      --config cleanup.policy=compact \
      --partitions 16 \
      --replication-factor 1 \
      --topic swh.test.objects.$object_type
done

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.content.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.directory.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.extid.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.metadata_authority.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.metadata_fetcher.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.origin.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.origin_visit.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.origin_visit_status.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.raw_extrinsic_metadata.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.release.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.revision.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.skipped_content.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.snapshot.


storage1 /opt/kafka/bin% TOPICS="release revision"

storage1 /opt/kafka/bin% for object_type in $(echo ${TOPICS}); do
    ./kafka-topics.sh --bootstrap-server $(hostname -f):9092 --create \
      --config cleanup.policy=compact \
      --partitions 16 \
      --replication-factor 1 \
      --topic swh.test.objects_privileged.$object_type
done

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects_privileged.release.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects_privileged.revision.
vsellier added a comment.Jun 3 2022, 3:16 PM

Permission of users should be ok:

  • mirror-test-rw
root@getty:~# diff -U3 /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh create_kafka_test_users_rw_rocquencourt_staging.sh 
--- /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh	2022-01-21 16:57:22.076322616 +0000
+++ create_kafka_test_users_rw_rocquencourt_staging.sh	2022-06-03 13:02:03.497371791 +0000
@@ -56,15 +72,15 @@
 	--entity-type users \
 	--entity-name $username
 
-topic_prefixes="swh.journal.objects. swh.journal.indexed."
+topic_prefixes="swh.test.objects."
 
 if [ $privileged = "privileged" ]; then
-	topic_prefixes="$topic_prefixes swh.journal.objects_privileged."
+	topic_prefixes="$topic_prefixes swh.test.objects_privileged."
 fi
 
 for topic_prefix in $topic_prefixes; do
 	echo "Granting access to topics $topic_prefix to $username"
-	for op in READ DESCRIBE; do
+	for op in READ DESCRIBE WRITE; do
 		/opt/kafka/bin/kafka-acls.sh --bootstrap-server $brokers --add --resource-pattern-type PREFIXED --topic $topic_prefix --allow-principal User:$username --operation $op
 	done
 done
root@getty:~# ./create_kafka_test_users_rw_rocquencourt_staging.sh --privileged mirror-test-rw
Creating user mirror-test-rw, with unprivileged access to consumer group prefix mirror-test-rw-
Password for user mirror-test-rw: 
Setting user credentials
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'mirror-test-rw'.
Granting access to topics swh.test.objects. to mirror-test-rw
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Granting access to topics swh.test.objects_privileged. to mirror-test-rw
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Granting access to consumer group prefix mirror-test-rw- to mirror-test-rw
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-rw-, patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-rw-, patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
  • mirror-test-ro
root@getty:~# diff -U3 /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh create_kafka_test_users_ro_rocquencourt_staging.sh 
--- /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh	2022-01-21 16:57:22.076322616 +0000
+++ create_kafka_test_users_ro_rocquencourt_staging.sh	2022-06-03 13:14:45.900009367 +0000
@@ -56,10 +56,10 @@
 	--entity-type users \
 	--entity-name $username
 
-topic_prefixes="swh.journal.objects. swh.journal.indexed."
+topic_prefixes="swh.test.objects."
 
 if [ $privileged = "privileged" ]; then
-	topic_prefixes="$topic_prefixes swh.journal.objects_privileged."
+	topic_prefixes="$topic_prefixes swh.test.objects_privileged."
 fi
 
 for topic_prefix in $topic_prefixes; do
root@getty:~# ./create_kafka_test_users_ro_rocquencourt_staging.sh --privileged mirror-test-ro
Creating user mirror-test-ro, with privileged access to consumer group prefix mirror-test-ro-
Password for user mirror-test-ro: 
Setting user credentials
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'mirror-test-ro'.
Granting access to topics swh.test.objects. to mirror-test-ro
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Granting access to topics swh.test.objects_privileged. to mirror-test-ro
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Granting access to consumer group prefix mirror-test-ro- to mirror-test-ro
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-ro-, patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-ro-, patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
vsellier closed this task as Resolved.Jun 3 2022, 3:17 PM
vsellier moved this task from Backlog to done on the System administration board.

Changing the status to resolved.
@douardda don't hesitate to reopen if it's not working as expected

vsellier reopened this task as Work in Progress.Jun 3 2022, 3:18 PM

heh, it misses the last part of the task (the commands to manage the topics)

vsellier added a comment.Jun 3 2022, 3:21 PM

btw the credentials are pushed in the credential store

vsellier added a comment.EditedJun 3 2022, 4:01 PM

I've also added the permissions to the user mirror-test-rw to create and destroy topics. So you should be able to manage the swh.test.objects[_privileged] topics lifecycle

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DELETE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DELETE, permissionType=ALLOW)
vsellier added a revision: D7962: kafka: add more options to the user management script.Jun 3 2022, 6:04 PM
douardda closed this task as Resolved.Jun 10 2022, 3:41 PM

I've added the DescribeConfigs and AlterConfigs permissions for the mirror-test-rw principal, now I can do all I need.

vsellier added a commit: rSPSITE395c4a2a79de: kafka: add more options to the user management script.Jun 16 2022, 5:49 PM
douardda mentioned this in T4336: Add deployment tests for the mirror.Jun 20 2022, 9:50 AM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 6:07 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal