Differential D3951

Switch production over to the new kafka cluster
ClosedPublic
Actions

Authored by olasd on Sep 15 2020, 11:39 AM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T2596: Switch main journal producer to new kafka cluster
T2597: Switch indexer journal client to new kafka cluster

Commits

rSPSITEdb2ebfd012b5: Anonymize messages sent to the production kafka cluster
rSPSITEab6794381e86: Use kafka[1-4].internal.softwareheritage.org as default kafka cluster

Summary

Use kafka[1-4].internal.softwareheritage.org as default kafka cluster
Anonymize messages sent to the production kafka cluster

Close T2596
Close T2597

Test Plan

octocatalog-diff on uffizi

*** Running octocatalog-diff on host uffizi.softwareheritage.org
I, [2020-09-15T11:29:26.724062 #36603]  INFO -- : Catalogs compiled for uffizi.softwareheritage.org
I, [2020-09-15T11:29:27.017178 #36603]  INFO -- : Diffs computed for uffizi.softwareheritage.org
diff origin/production/uffizi.softwareheritage.org current/uffizi.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/storage/storage.yml] =>
   parameters =>
     content =>
      @@ -95,9 +95,11 @@
             args:
               brokers:
      -        - esnode1.internal.softwareheritage.org
      -        - esnode2.internal.softwareheritage.org
      -        - esnode3.internal.softwareheritage.org
      +        - kafka1.internal.softwareheritage.org
      +        - kafka2.internal.softwareheritage.org
      +        - kafka3.internal.softwareheritage.org
      +        - kafka4.internal.softwareheritage.org
               prefix: swh.journal.objects
               client_id: swh.storage.journal_writer.uffizi
      +        anonymize: true
               producer_config:
                 message.max.bytes: 1000000000
*******************************************
*** End octocatalog-diff on uffizi.softwareheritage.org

octocatalog-diff on getty:

*** Running octocatalog-diff on host getty.internal.softwareheritage.org
I, [2020-09-15T11:31:50.224834 #38835]  INFO -- : Catalogs compiled for getty.internal.softwareheritage.org
I, [2020-09-15T11:31:50.427346 #38835]  INFO -- : Diffs computed for getty.internal.softwareheritage.org
diff origin/production/getty.internal.softwareheritage.org current/getty.internal.softwareheritage.org
*******************************************
  File[/etc/softwareheritage/indexer/journal_client.yml] =>
   parameters =>
     content =>
      @@ -2,7 +2,8 @@
       journal:
         brokers:
      -  - esnode1.internal.softwareheritage.org
      -  - esnode2.internal.softwareheritage.org
      -  - esnode3.internal.softwareheritage.org
      +  - kafka1.internal.softwareheritage.org
      +  - kafka2.internal.softwareheritage.org
      +  - kafka3.internal.softwareheritage.org
      +  - kafka4.internal.softwareheritage.org
         group_id: swh.indexer.journal_client
       scheduler:
*******************************************
*** End octocatalog-diff on getty.internal.softwareheritage.org