Page MenuHomeSoftware Heritage

client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
ClosedPublic

Authored by douardda on May 5 2020, 4:39 PM.

Details

Summary

Remove this global variable and check, at instanciation time, that subscribed
topics exist on the kafka broker.

The semantics of the `object_types` argument is thus a bit different:
if unset (None), theclient will subscribe to the whole prefix (namely it will
subscribe to "^{prefix}[.]", so any topic "under" the prefix).

Diff Detail

Repository
rDJNL Journal infrastructure
Branch
better-topics-management
Lint
Lint Skipped
Unit
Unit Tests Skipped
Build Status
Buildable 12274
Build 18616: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 18615: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D3126 (id=11100)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit 1c2e855bc958b1f5571703e4d750666deae3db7c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), theclient will subscribe to the whole prefix (namely it will
    subscribe to "^{prefix}[.]", so any topic "under" the prefix).

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/63/ for more details.

few changes according to comments on IRC

  • do not use a regex subscription, subscribe to all existing topics instead
  • docstring
  • replace a list comp by any()
  • put back the subscribe() method, it seems to be used, somewhere in the universe.

Build is green

Patch application report for D3126 (id=11101)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit f553e83dd139092524875c9a4de5c0a40500f61c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/64/ for more details.

ardumont added a subscriber: ardumont.

lgtm!

Thanks ;)

This revision is now accepted and ready to land.May 5 2020, 6:14 PM

Build is green

Patch application report for D3126 (id=11116)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit ef52b54615b14577d29e8a6257184f72b5df01f2
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/65/ for more details.

Build is green

Patch application report for D3126 (id=11119)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit 8ea0de47a79c5b18c555576b64997d8dc81a2071
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instantiation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/67/ for more details.