Page MenuHomeSoftware Heritage

Create test topics on staging's kafka for the mirror tests
Closed, ResolvedPublic

Description

  • Create the usual swh topics under the swh.test.journal... namespace on the staging's kafka with 16 partitions.
  • Provide 2 users:
    • mirror-test-rw for dataset intialisation
    • mirror-test-ro to run the tests
  • Provide a set a command lines to manage the topics (reset/recreate/....)

Event Timeline

vsellier triaged this task as Normal priority.May 23 2022, 1:48 PM
vsellier created this task.

topics to be created under the `swh.test' prefix (or similar).

thanks, I forgot to mention that

vsellier changed the task status from Open to Work in Progress.Jun 2 2022, 10:19 AM

Topics created:

storage1 /opt/kafka/bin% TOPICS="content directory extid metadata_authority metadata_fetcher origin origin_visit origin_visit_status raw_extrinsic_metadata release revision skipped_content snapshot"

storage1 /opt/kafka/bin% for object_type in $(echo ${TOPICS}); do
    ./kafka-topics.sh --bootstrap-server $(hostname -f):9092 --create \
      --config cleanup.policy=compact \
      --partitions 16 \
      --replication-factor 1 \
      --topic swh.test.objects.$object_type
done

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.content.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.directory.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.extid.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.metadata_authority.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.metadata_fetcher.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.origin.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.origin_visit.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.origin_visit_status.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.raw_extrinsic_metadata.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.release.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.revision.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.skipped_content.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects.snapshot.


storage1 /opt/kafka/bin% TOPICS="release revision"

storage1 /opt/kafka/bin% for object_type in $(echo ${TOPICS}); do
    ./kafka-topics.sh --bootstrap-server $(hostname -f):9092 --create \
      --config cleanup.policy=compact \
      --partitions 16 \
      --replication-factor 1 \
      --topic swh.test.objects_privileged.$object_type
done

WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects_privileged.release.
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic swh.test.objects_privileged.revision.

Permission of users should be ok:

  • mirror-test-rw
root@getty:~# diff -U3 /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh create_kafka_test_users_rw_rocquencourt_staging.sh 
--- /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh	2022-01-21 16:57:22.076322616 +0000
+++ create_kafka_test_users_rw_rocquencourt_staging.sh	2022-06-03 13:02:03.497371791 +0000
@@ -56,15 +72,15 @@
 	--entity-type users \
 	--entity-name $username
 
-topic_prefixes="swh.journal.objects. swh.journal.indexed."
+topic_prefixes="swh.test.objects."
 
 if [ $privileged = "privileged" ]; then
-	topic_prefixes="$topic_prefixes swh.journal.objects_privileged."
+	topic_prefixes="$topic_prefixes swh.test.objects_privileged."
 fi
 
 for topic_prefix in $topic_prefixes; do
 	echo "Granting access to topics $topic_prefix to $username"
-	for op in READ DESCRIBE; do
+	for op in READ DESCRIBE WRITE; do
 		/opt/kafka/bin/kafka-acls.sh --bootstrap-server $brokers --add --resource-pattern-type PREFIXED --topic $topic_prefix --allow-principal User:$username --operation $op
 	done
 done
root@getty:~# ./create_kafka_test_users_rw_rocquencourt_staging.sh --privileged mirror-test-rw
Creating user mirror-test-rw, with unprivileged access to consumer group prefix mirror-test-rw-
Password for user mirror-test-rw: 
Setting user credentials
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'mirror-test-rw'.
Granting access to topics swh.test.objects. to mirror-test-rw
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Granting access to topics swh.test.objects_privileged. to mirror-test-rw
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Granting access to consumer group prefix mirror-test-rw- to mirror-test-rw
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-rw-, patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-rw-, patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
  • mirror-test-ro
root@getty:~# diff -U3 /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh create_kafka_test_users_ro_rocquencourt_staging.sh 
--- /usr/local/sbin/create_kafka_users_rocquencourt_staging.sh	2022-01-21 16:57:22.076322616 +0000
+++ create_kafka_test_users_ro_rocquencourt_staging.sh	2022-06-03 13:14:45.900009367 +0000
@@ -56,10 +56,10 @@
 	--entity-type users \
 	--entity-name $username
 
-topic_prefixes="swh.journal.objects. swh.journal.indexed."
+topic_prefixes="swh.test.objects."
 
 if [ $privileged = "privileged" ]; then
-	topic_prefixes="$topic_prefixes swh.journal.objects_privileged."
+	topic_prefixes="$topic_prefixes swh.test.objects_privileged."
 fi
 
 for topic_prefix in $topic_prefixes; do
root@getty:~# ./create_kafka_test_users_ro_rocquencourt_staging.sh --privileged mirror-test-ro
Creating user mirror-test-ro, with privileged access to consumer group prefix mirror-test-ro-
Password for user mirror-test-ro: 
Setting user credentials
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'mirror-test-ro'.
Granting access to topics swh.test.objects. to mirror-test-ro
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Granting access to topics swh.test.objects_privileged. to mirror-test-ro
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW) 

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Granting access to consumer group prefix mirror-test-ro- to mirror-test-ro
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-ro-, patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=mirror-test-ro-, patternType=PREFIXED)`: 
 	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
vsellier moved this task from Backlog to done on the System administration board.

Changing the status to resolved.
@douardda don't hesitate to reopen if it's not working as expected

vsellier reopened this task as Work in Progress.Jun 3 2022, 3:18 PM

heh, it misses the last part of the task (the commands to manage the topics)

btw the credentials are pushed in the credential store

I've also added the permissions to the user mirror-test-rw to create and destroy topics. So you should be able to manage the swh.test.objects[_privileged] topics lifecycle

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DELETE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.test.objects_privileged., patternType=PREFIXED)`: 
 	(principal=User:mirror-test-rw, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:mirror-test-ro, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:mirror-test-rw, host=*, operation=DELETE, permissionType=ALLOW)

I've added the DescribeConfigs and AlterConfigs permissions for the mirror-test-rw principal, now I can do all I need.