Differential D4620

staging: configure idx-storage to write to kafka
ClosedPublic
Actions

Authored by vsellier on Nov 27 2020, 10:43 AM.

Details

Reviewers

ardumont

Group Reviewers

System administrators

Maniphest Tasks

T2816: Enable the journal-writer for the swh-idx-storage in staging

Commits

rSPSITEa2a84c2efb3e: staging: configure idx-storage to write to kafka

Summary

The topic name was temporary chosen for staging according to the T2780
suggestion.
The partitioning, retention, and cleanup policy will have
to be defined before landing this in production.

Related to T2816

Test Plan

octocatalog-diff

storage1.staging

diff origin/production/storage1.internal.staging.swh.network current/storage1.internal.staging.swh.network
*******************************************
  File[/etc/softwareheritage/storage/indexer.yml] =>
   parameters =>
     content =>
      @@ -4,3 +4,11 @@
         db: host=db1.internal.staging.swh.network port=5432 user=swh-indexer dbname=swh-indexer
           password=swh::deploy::indexer::storage::db::password
      +  journal_writer:
      +    cls: kafka
      +    brokers:
      +    - journal0.internal.staging.swh.network
      +    prefix: swh.journal.indexed
      +    client_id: swh.idx_storage.journal_writer.storage1
      +    producer_config:
      +      message.max.bytes: 1000000000
      _
*******************************************
*** End octocatalog-diff on storage1.internal.staging.swh.network

No changes in production :

*** Running octocatalog-diff on host saam.internal.softwareheritage.org
I, [2020-11-27T10:32:46.714932 #7376]  INFO -- : Catalogs compiled for saam.internal.softwareheritage.org
I, [2020-11-27T10:32:48.136707 #7376]  INFO -- : Diffs computed for saam.internal.softwareheritage.org
I, [2020-11-27T10:32:48.136747 #7376]  INFO -- : No differences
*** End octocatalog-diff on saam.internal.softwareheritage.org