Page MenuHomeSoftware Heritage
Differential D8492

indexer: Allow journal client authentication configuration
ClosedPublic
Actions

Authored by ardumont on Sep 15 2022, 5:49 PM.

Details

Reviewers
vsellier
Group Reviewers
System administrators
Maniphest Tasks
T4459: Deploy swh-indexer > v2.6 on staging then production
Commits
rSPSITE009c924339c1: indexer: Allow journal client authentication configuration
Summary

This extends the current production indexer journal clients to use their credentials and
avoid passing through the vpn.

This does not touch the staging workers configuration though.

This got adapted according to the documentation [1].

Related to T4459
[1] https://docs.softwareheritage.org/devel/swh-journal/journal-clients.html#journal-client-authentication

Test Plan
$ $SWH_PUPPET_ENVIRONMENT_HOME/bin/octocatalog-diff indexer-worker04.euwest.azure.internal.softwareheritage.org
diff origin/production/indexer-worker04.euwest.azure.internal.softwareheritage.org current/indexer-worker04.euwest.azure.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/indexer/extrinsic_metadata.yml] =>
   parameters =>
     content =>
      @@ -100,6 +100,10 @@
         - kafka3.internal.softwareheritage.org
         - kafka4.internal.softwareheritage.org
      -  group_id: swh.indexer.journal_client.extrinsic_metadata
      +  group_id: swh::deploy::indexer_journal_client::journal::username-swh.indexer.journal_client.extrinsic_metadata
         prefix: swh.journal.objects
      +  sasl.mechanism: SCRAM-SHA-512
      +  security.protocol: SASL_SSL
      +  sasl.username: swh::deploy::indexer_journal_client::journal::username
      +  sasl.password: swh-deploy-indexer_journal_client-journal-password
       tools:
         name: swh-metadata-detector
*******************************************
  File[/etc/softwareheritage/indexer/origin_intrinsic_metadata.yml] =>
   parameters =>
     content =>
      @@ -100,7 +100,11 @@
         - kafka3.internal.softwareheritage.org
         - kafka4.internal.softwareheritage.org
      -  group_id: swh.indexer.journal_client.origin_intrinsic_metadata
      +  group_id: swh::deploy::indexer_journal_client::journal::username-swh.indexer.journal_client.origin_intrinsic_metadata
         prefix: swh.journal.objects
         batch_size: 200
      +  sasl.mechanism: SCRAM-SHA-512
      +  security.protocol: SASL_SSL
      +  sasl.username: swh::deploy::indexer_journal_client::journal::username
      +  sasl.password: swh-deploy-indexer_journal_client-journal-password
       tools:
         name: swh-metadata-detector
*******************************************
*** End octocatalog-diff on indexer-worker04.euwest.azure.internal.softwareheritage.org

staging: noop

*** Running octocatalog-diff on host worker0.internal.staging.swh.network
I, [2022-09-15T17:49:45.747441 #3456278]  INFO -- : Catalogs compiled for worker0.internal.staging.swh.network
I, [2022-09-15T17:49:46.142017 #3456278]  INFO -- : Diffs computed for worker0.internal.staging.swh.network
I, [2022-09-15T17:49:46.142058 #3456278]  INFO -- : No differences
*** End octocatalog-diff on worker0.internal.staging.swh.network

Diff Detail

Repository
rSPSITE puppet-swh-site
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Sep 15 2022, 5:49 PM
Herald added a reviewer: System administrators. · View Herald TranscriptSep 15 2022, 5:49 PM
Harbormaster completed remote builds in B31573: Diff 30581.Sep 15 2022, 5:49 PM
ardumont requested review of this revision.Sep 15 2022, 5:49 PM
ardumont edited the test plan for this revision. (Show Details)Sep 15 2022, 5:50 PM
ardumont edited the summary of this revision. (Show Details)
vsellier accepted this revision.Sep 15 2022, 5:55 PM
This revision is now accepted and ready to land.Sep 15 2022, 5:55 PM
vsellier requested changes to this revision.Sep 15 2022, 5:56 PM
This revision now requires changes to proceed.Sep 15 2022, 5:56 PM
vsellier added a comment.EditedSep 15 2022, 5:58 PM

The group id of the authenticated consumers have to be probably updated to match the kafka acls

ardumont added a comment.Sep 15 2022, 6:10 PM

The group id of the authenticated consumers have to be probably updated to match the kafka acls

mmm, yeah, right, i had forgotten that subtlety.

ardumont edited the test plan for this revision. (Show Details)Sep 15 2022, 6:10 PM
ardumont updated this revision to Diff 30582.EditedSep 15 2022, 6:11 PM

Adapt according to review

Harbormaster completed remote builds in B31574: Diff 30582.Sep 15 2022, 6:11 PM
ardumont mentioned this in T4459: Deploy swh-indexer > v2.6 on staging then production.Sep 15 2022, 6:15 PM
ardumont added a child revision: D8493: indexer: Use public brokers in production, internal ones for staging.Sep 15 2022, 6:28 PM
vsellier accepted this revision.Sep 16 2022, 3:52 PM

perhaps a little concern regarding the length of the group_id but nothing blocking

This revision is now accepted and ready to land.Sep 16 2022, 3:52 PM
ardumont updated this revision to Diff 30595.Sep 16 2022, 4:46 PM

Rebase

Harbormaster completed remote builds in B31587: Diff 30595.Sep 16 2022, 4:46 PM
Closed by commit rSPSITE009c924339c1: indexer: Allow journal client authentication configuration (authored by ardumont). · Explain WhySep 16 2022, 4:47 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rSPSITE009c924339c1: indexer: Allow journal client authentication configuration.

Revision Contents
Changeset List

PathSize
data/
common/
5 lines
deployments/
staging/
5 lines
site-modules/
profile/
manifests/
swh/
deploy/
25 lines

Diff 30596

View Options

data/common/common.yaml

Loading...
View Options

data/deployments/staging/common.yaml

Loading...
View Options

site-modules/profile/manifests/swh/deploy/indexer_journal_client.pp

Loading...
About · Developer information · Legal