Page MenuHomeSoftware Heritage

indexer: Allow journal client authentication configuration
ClosedPublic

Authored by ardumont on Sep 15 2022, 5:49 PM.

Details

Summary

This extends the current production indexer journal clients to use their credentials and
avoid passing through the vpn.

This does not touch the staging workers configuration though.

This got adapted according to the documentation [1].

Related to T4459
[1] https://docs.softwareheritage.org/devel/swh-journal/journal-clients.html#journal-client-authentication

Test Plan
$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff indexer-worker04.euwest.azure.internal.softwareheritage.org
diff origin/production/indexer-worker04.euwest.azure.internal.softwareheritage.org current/indexer-worker04.euwest.azure.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/indexer/extrinsic_metadata.yml] =>
   parameters =>
     content =>
      @@ -100,6 +100,10 @@
         - kafka3.internal.softwareheritage.org
         - kafka4.internal.softwareheritage.org
      -  group_id: swh.indexer.journal_client.extrinsic_metadata
      +  group_id: swh::deploy::indexer_journal_client::journal::username-swh.indexer.journal_client.extrinsic_metadata
         prefix: swh.journal.objects
      +  sasl.mechanism: SCRAM-SHA-512
      +  security.protocol: SASL_SSL
      +  sasl.username: swh::deploy::indexer_journal_client::journal::username
      +  sasl.password: swh-deploy-indexer_journal_client-journal-password
       tools:
         name: swh-metadata-detector
*******************************************
  File[/etc/softwareheritage/indexer/origin_intrinsic_metadata.yml] =>
   parameters =>
     content =>
      @@ -100,7 +100,11 @@
         - kafka3.internal.softwareheritage.org
         - kafka4.internal.softwareheritage.org
      -  group_id: swh.indexer.journal_client.origin_intrinsic_metadata
      +  group_id: swh::deploy::indexer_journal_client::journal::username-swh.indexer.journal_client.origin_intrinsic_metadata
         prefix: swh.journal.objects
         batch_size: 200
      +  sasl.mechanism: SCRAM-SHA-512
      +  security.protocol: SASL_SSL
      +  sasl.username: swh::deploy::indexer_journal_client::journal::username
      +  sasl.password: swh-deploy-indexer_journal_client-journal-password
       tools:
         name: swh-metadata-detector
*******************************************
*** End octocatalog-diff on indexer-worker04.euwest.azure.internal.softwareheritage.org

staging: noop

*** Running octocatalog-diff on host worker0.internal.staging.swh.network
I, [2022-09-15T17:49:45.747441 #3456278]  INFO -- : Catalogs compiled for worker0.internal.staging.swh.network
I, [2022-09-15T17:49:46.142017 #3456278]  INFO -- : Diffs computed for worker0.internal.staging.swh.network
I, [2022-09-15T17:49:46.142058 #3456278]  INFO -- : No differences
*** End octocatalog-diff on worker0.internal.staging.swh.network

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont edited the summary of this revision. (Show Details)
This revision is now accepted and ready to land.Sep 15 2022, 5:55 PM
This revision now requires changes to proceed.Sep 15 2022, 5:56 PM

The group id of the authenticated consumers have to be probably updated to match the kafka acls

The group id of the authenticated consumers have to be probably updated to match the kafka acls

mmm, yeah, right, i had forgotten that subtlety.

Adapt according to review

perhaps a little concern regarding the length of the group_id but nothing blocking

This revision is now accepted and ready to land.Sep 16 2022, 3:52 PM