Page MenuHomeSoftware Heritage

client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
ClosedPublic

Authored by douardda on Tue, May 5, 4:39 PM.

Details

Summary

Remove this global variable and check, at instanciation time, that subscribed
topics exist on the kafka broker.

The semantics of the `object_types` argument is thus a bit different:
if unset (None), theclient will subscribe to the whole prefix (namely it will
subscribe to "^{prefix}[.]", so any topic "under" the prefix).

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Tue, May 5, 4:39 PM

Build is green

Patch application report for D3126 (id=11100)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit 1c2e855bc958b1f5571703e4d750666deae3db7c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), theclient will subscribe to the whole prefix (namely it will
    subscribe to "^{prefix}[.]", so any topic "under" the prefix).

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/63/ for more details.

douardda updated this revision to Diff 11101.Tue, May 5, 5:11 PM

few changes according to comments on IRC

  • do not use a regex subscription, subscribe to all existing topics instead
  • docstring
  • replace a list comp by any()
  • put back the subscribe() method, it seems to be used, somewhere in the universe.

Build is green

Patch application report for D3126 (id=11101)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit f553e83dd139092524875c9a4de5c0a40500f61c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/64/ for more details.

ardumont accepted this revision.Tue, May 5, 6:14 PM
ardumont added a subscriber: ardumont.

lgtm!

Thanks ;)

This revision is now accepted and ready to land.Tue, May 5, 6:14 PM
douardda updated this revision to Diff 11116.Wed, May 6, 5:34 PM

fix a few typos

Build is green

Patch application report for D3126 (id=11116)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit ef52b54615b14577d29e8a6257184f72b5df01f2
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/65/ for more details.

douardda updated this revision to Diff 11119.Thu, May 7, 9:25 AM

typo in commit message

Build is green

Patch application report for D3126 (id=11119)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit 8ea0de47a79c5b18c555576b64997d8dc81a2071
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instantiation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/67/ for more details.