Page MenuHomeSoftware Heritage

Add support for the rdkafka 'stats_cb' config option in JournalClient
ClosedPublic

Authored by douardda on Jan 6 2022, 12:16 PM.

Details

Summary

this options allows to define a callback which will be called once every
'statistics.interval.ms' ms by rdkafka with a bunch of statistics (as a
json string).

See https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md

This allows to define this callback as a string of the form:

"path.to.module:function"

Depends on D6882

Test Plan

WIP because needs tests

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

Build is green

Patch application report for D6884 (id=24961)

Could not rebase; Attempt merge onto 4e5e00988a...

Updating 4e5e009..d877377
Fast-forward
 swh/journal/client.py | 54 +++++++++++++++++++++++++++++++++------------------
 1 file changed, 35 insertions(+), 19 deletions(-)
Changes applied before test
commit d87737790bc067a30510e87c20b402d5d310f2dc
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jan 6 12:07:50 2022 +0100

    Add support for the rdkafka 'stats_cb' config option in JournalClient
    
    this options allows to define a callback which will be called once every
    'statistics.interval.ms' ms by rdkafka with a bunch of statistics (as a
    json string).
    
    See https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md
    
    This allows to define this callback as a string of the form:
    
      "path.to.module:function"

commit 35bb07d74b2bc19353280770f825648c5d053bc4
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Jan 4 17:25:02 2022 +0100

    Remove 'process_timeout' from JournalClient's arguments
    
    it was not used anywhere, so simplify the code a bit.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/189/ for more details.

move the code in get_journal_client

Build is green

Patch application report for D6884 (id=24966)

Could not rebase; Attempt merge onto 4e5e00988a...

Updating 4e5e009..5b17c50
Fast-forward
 swh/journal/client.py | 53 +++++++++++++++++++++++++++++++++------------------
 1 file changed, 34 insertions(+), 19 deletions(-)
Changes applied before test
commit 5b17c50d32809c1cb4f3505846d23572c5650496
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Jan 6 12:07:50 2022 +0100

    Add support for the rdkafka 'stats_cb' config option in get_journal_client
    
    this options allows to define a callback which will be called once every
    'statistics.interval.ms' ms by rdkafka with a bunch of statistics (as a
    json string).
    
    See https://github.com/edenhill/librdkafka/blob/master/STATISTICS.md
    
    This allows to define this callback as a string of the form:
    
      "path.to.module:function"

commit 0d115993e0f1bd5e049ba07a1185ae29fcb6e654
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Jan 4 17:25:02 2022 +0100

    Remove 'process_timeout' from JournalClient's arguments
    
    it was not used anywhere, so simplify the code a bit.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/191/ for more details.

douardda retitled this revision from [WIP] Add support for the rdkafka 'stats_cb' config option in JournalClient to Add support for the rdkafka 'stats_cb' config option in JournalClient.Jan 14 2022, 10:52 AM
olasd added a subscriber: olasd.

I'm kinda wondering if this import stuff should move to a common module - I think we do kind of the same thing with entrypoints?

This revision is now accepted and ready to land.Jan 18 2022, 1:42 PM
In D6884#181115, @olasd wrote:

I'm kinda wondering if this import stuff should move to a common module - I think we do kind of the same thing with entrypoints?

yeah we might think of it eventually.