HomeSoftware Heritage

Make the kafka group_id prefix configurable in the config file

Description

Make the kafka group_id prefix configurable in the config file

rather than hardcoding it to 'swh-dataset-export-', use the 'group_id'
value from the 'journal' section of the config file as prefix, if given
9otherwise default to the former value).

This is needed because current auth policy of swh kafka cluster only allow
group_id to start with the actual login for authenticated connection.
So we need to be able to specify this group_id prefix.

Details

Provenance
douarddaAuthored on Dec 15 2021, 4:50 PM
douarddaPushed on Mar 29 2022, 5:24 PM
Differential Revision
D7382: Make the kafka group_id prefix configurable in the config file
Parents
rDDATASET769b6a77d250: Use a named logger for journalprocessor.py
Branches
Unknown
Tags
Unknown
Build Status
Buildable 27952
Build 43772: test-and-buildJenkins console · Jenkins