Page MenuHomeSoftware Heritage
Create Task
Maniphest T2682

Deploy a small publicly available kafka server (with some content) on a staging (+ the related objstorage)
Closed, MigratedEdits Locked
Actions

Description

The idea is that I'd like to be able to document the mirror stack so that it "just works" out of the box so people interested in setting up a mirror can test it.

For this, we need a small kafka server with some data in it (say one or 2 small gitab/gitea ingested instances) that is publicly available. Only anonymized topics should be accessible there.

For the content-replayer to work, it will also need an objectstorage, since the content replayer pulls blobs from one objstorage to another (reading sha1s from the content topic).
This might be a bit trickier if we want to make it "read-only" to prevent kiddies from playing with it...
[edit] the ReadObjStorageFilter is our friend here[/edit]

It might also be useful to have the kafka accessible with and without authentication (to test that authentication layer), but it's not that important.

Actions :

  • diff landed and applied on the server
  • VIP 128.93.166.40 configured on the firewall
  • NAT Port forward of port 9093 from public ip to internal journal0 declared on the firewall
  • DNS declaration of broker0.journal.staging.swh.network in gandi
  • Ask to DSI to apply the kafka firewall profile to 128.93.166.40
  • Configure a user to test the pipeline
  • Configure a read-only object storage on webapp.staging

Revisions and Commits

rSENV Puppet Environment
D4775rSENV661482496784 Add objstorage0.staging.swh.network node to expose a r/o objstorage node
rSPSITE puppet-swh-site
D4776rSPSITE96bfcf42004d [staging] Configure and expose to internet a read-only objstorage
D4726rSPSITE5c693c5cf08b kafka: activate the authentication on the public network

Related Objects
Search...

Event Timeline

douardda triaged this task as High priority.Oct 9 2020, 3:37 PM
douardda created this task.
douardda updated the task description. (Show Details)Oct 9 2020, 3:47 PM
vsellier mentioned this in T2790: [staging] deploy the journal infrastructure.Nov 17 2020, 2:52 PM
vsellier added a subtask: T2790: [staging] deploy the journal infrastructure.
vsellier added a project: Staging environment.
vsellier changed the status of subtask T2790: [staging] deploy the journal infrastructure from Open to Work in Progress.Nov 17 2020, 3:09 PM
vsellier closed subtask T2790: [staging] deploy the journal infrastructure as Resolved.Nov 30 2020, 10:47 AM
vsellier changed the task status from Open to Work in Progress.Dec 10 2020, 5:41 PM
vsellier added a revision: D4726: kafka: activate the authentication on the public network.Dec 11 2020, 3:17 PM
vsellier mentioned this in rSENV84531f26646d: Upgrade the journal0 to have the first CN matching broker1.journal.staging.swh..Dec 11 2020, 4:24 PM
vsellier added a commit: rSPSITE5c693c5cf08b: kafka: activate the authentication on the public network.Dec 11 2020, 5:19 PM
vsellier added a comment.EditedDec 11 2020, 6:11 PM
  • diff landed and applied on the server
  • VIP 128.93.166.40 configured on the firewall
  • NAT Port forward of port 9093 from public ip to internal journal0 declared on the firewall
  • DNS declaration of broker0.journal.staging.swh.network in gandi
  • Ask to DSI to apply the kafka firewall profile to 128.93.166.40
  • Configure a user to test the pipeline

    Note copied to the task description, please don't edit here
vsellier added a comment.Dec 18 2020, 4:57 PM

The request to expose the journal to internet was done this afternoon to the dsi.

vsellier updated the task description. (Show Details)Dec 18 2020, 4:58 PM
vsellier updated the task description. (Show Details)Dec 21 2020, 12:24 PM

The network configuration is done. The server is now accessible from the internet at broker0.journal.staging.swh.network:9093

vsellier updated the task description. (Show Details)Dec 21 2020, 12:38 PM
vsellier added a comment.Dec 21 2020, 12:57 PM

A user was correctly configured and a read test performed :

  • User creation:
% /opt/kafka/bin/kafka-configs.sh \
    --zookeeper journal0.internal.staging.swh.network:2181/kafka/softwareheritage \
    --alter \
    --add-config 'SCRAM-SHA-256=[iterations=8192,password=xxxx],SCRAM-SHA-512=[password=xxxx]' \
    --entity-type users \
    --entity-name swh-test
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'swh-test'.
  • acl configuration
bootstrap_servers=journal0.internal.staging.swh.network:9092
username=swh-test

# Allow READ and DESCRIBE on unprivileged topics
% /opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation READ

Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-test, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-test, host=*, operation=READ, permissionType=ALLOW) 

% /opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:$username --operation DESCRIBE
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-test, host=*, operation=DESCRIBE, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-test, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:swh-test, host=*, operation=DESCRIBE, permissionType=ALLOW) 

# Allow READ on consumer groups prefixed with `$username-`
%  /opt/kafka/bin/kafka-acls.sh --bootstrap-server $bootstrap_servers --add --resource-pattern-type PREFIXED --group ${username}- --allow-principal User:$username --operation READ
Adding ACLs for resource `ResourcePattern(resourceType=GROUP, name=swh-test-, patternType=PREFIXED)`: 
 	(principal=User:swh-test, host=*, operation=READ, permissionType=ALLOW) 

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=swh-test-, patternType=PREFIXED)`: 
 	(principal=User:swh-test, host=*, operation=READ, permissionType=ALLOW)
  • Read test from internet :
~/Downloads/kafka_2.13-2.6.0/bin ❯ ./kafka-console-consumer.sh --consumer.config=kafka.properties --bootstrap-server  broker0.journal.staging.swh.network:9093  --topic swh.journal.objects.release --group swh-test-test --from-beginning
...

Processed a total of 57000 messages
% ./kafka-consumer-groups.sh --bootstrap-server ${SERVER} --describe --all-topics --group swh-test-test 

Consumer group 'swh-test-test' has no active members.

GROUP           TOPIC                       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
swh-test-test   swh.journal.objects.release 10         1575            2339            764             -               -               -
swh-test-test   swh.journal.objects.release 43         1452            2150            698             -               -               -
swh-test-test   swh.journal.objects.release 6          1543            2290            747             -               -               -
...
vsellier updated the task description. (Show Details)Dec 21 2020, 12:58 PM
vsellier added a revision: D4775: Add objstorage0.staging.swh.network node to expose a r/o objstorage node.Dec 21 2020, 4:48 PM
vsellier added a revision: D4776: [staging] Configure and expose to internet a read-only objstorage.Dec 21 2020, 6:01 PM
vsellier added a commit: rSENV661482496784: Add objstorage0.staging.swh.network node to expose a r/o objstorage node.Dec 21 2020, 6:18 PM
vsellier added a commit: rSPSITE96bfcf42004d: [staging] Configure and expose to internet a read-only objstorage.Dec 21 2020, 6:20 PM
vsellier mentioned this in rSPRE6efec4e443b4: staging: Add objstorage0 node.Dec 21 2020, 6:51 PM
vsellier added a comment.EditedDec 21 2020, 7:32 PM
  • A new vm objstorage0.internal.staging.swh.network is configured with an read-only object storage service
  • It's exposed to internet via the reverse proxy at https://objstorage.staging.swh.network (it quite different as the usual objstorage:5003 url but it allow to expose the service without new network configuration)
  • DNS entry added on gandi
  • Inventory updated
vsellier updated the task description. (Show Details)Dec 22 2020, 9:54 AM

Everything looks good, let's try to add some documentation before closing the issue

vsellier closed this task as Resolved.EditedJan 4 2021, 12:33 PM

Closing this task as all the direct work is done.
The documentation will be addressed in T2920

vsellier mentioned this in T3621: Create a production read-only objstorage.Sep 29 2021, 5:39 PM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 5:58 PM
gitlab-migration claimed this task.
gitlab-migration added subscribers: vsellier, gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T2790: [staging] deploy the journal infrastructure from Resolved to Migrated.Oct 19 2022, 6:00 PM
About · Developer information · Legal