Page MenuHomeSoftware Heritage
Differential D3126

client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
ClosedPublic
Actions

Authored by douardda on May 5 2020, 4:39 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDJNL8ea0de47a79c: client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
Summary

Remove this global variable and check, at instanciation time, that subscribed
topics exist on the kafka broker.

The semantics of the `object_types` argument is thus a bit different:
if unset (None), theclient will subscribe to the whole prefix (namely it will
subscribe to "^{prefix}[.]", so any topic "under" the prefix).

Diff Detail

Repository
rDJNL Journal infrastructure
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.May 5 2020, 4:39 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 5 2020, 4:39 PM
douardda mentioned this in D3119: Add skipped_content to the list of accepted objects.May 5 2020, 4:40 PM
swh-public-ci added a comment.May 5 2020, 4:40 PM

Build is green

Patch application report for D3126 (id=11100)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit 1c2e855bc958b1f5571703e4d750666deae3db7c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), theclient will subscribe to the whole prefix (namely it will
    subscribe to "^{prefix}[.]", so any topic "under" the prefix).

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/63/ for more details.

Harbormaster completed remote builds in B12273: Diff 11100.May 5 2020, 4:40 PM
douardda updated this revision to Diff 11101.May 5 2020, 5:11 PM

few changes according to comments on IRC

  • do not use a regex subscription, subscribe to all existing topics instead
  • docstring
  • replace a list comp by any()
  • put back the subscribe() method, it seems to be used, somewhere in the universe.
swh-public-ci added a comment.May 5 2020, 5:13 PM

Build is green

Patch application report for D3126 (id=11101)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit f553e83dd139092524875c9a4de5c0a40500f61c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/64/ for more details.

Harbormaster completed remote builds in B12274: Diff 11101.May 5 2020, 5:13 PM
ardumont accepted this revision.May 5 2020, 6:14 PM
ardumont added a subscriber: ardumont.

lgtm!

Thanks ;)

This revision is now accepted and ready to land.May 5 2020, 6:14 PM
douardda updated this revision to Diff 11116.May 6 2020, 5:34 PM

fix a few typos

swh-public-ci added a comment.May 6 2020, 5:36 PM

Build is green

Patch application report for D3126 (id=11116)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit ef52b54615b14577d29e8a6257184f72b5df01f2
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instanciation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/65/ for more details.

Harbormaster completed remote builds in B12289: Diff 11116.May 6 2020, 5:36 PM
douardda added a child revision: D3130: Refactor the pytest_plugin.May 6 2020, 5:40 PM
douardda updated this revision to Diff 11119.May 7 2020, 9:25 AM

typo in commit message

swh-public-ci added a comment.May 7 2020, 9:27 AM

Build is green

Patch application report for D3126 (id=11119)

Rebasing onto fa9ab16c30...

Current branch diff-target is up to date.
Changes applied before test
commit 8ea0de47a79c5b18c555576b64997d8dc81a2071
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue May 5 16:17:24 2020 +0200

    client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection
    
    Remove this global variable and check, at instantiation time, that subscribed
    topics exist on the kafka broker.
    
    The semantics of the ``object_types`` argument is thus a bit different:
    if unset (None), the client will subscribe to any existing topic on the kafka
    broker that starts with the prefix.

See https://jenkins.softwareheritage.org/job/DJNL/job/tests-on-diff/67/ for more details.

Harbormaster completed remote builds in B12292: Diff 11119.May 7 2020, 9:27 AM
Closed by commit rDJNL8ea0de47a79c: client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection (authored by douardda). · Explain WhyMay 7 2020, 9:33 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDJNL8ea0de47a79c: client: replace the hardcoded ACCEPTED_OBJECT_TYPES by kafka introspection.

Revision Contents
Changeset List

PathSize
swh/
journal/
66 lines
tests/
100 lines
4 lines

Diff 11121

View Options

swh/journal/client.py

Loading...
View Options

swh/journal/tests/test_client.py

Loading...
View Options

swh/journal/tests/utils.py

Loading...
About · Developer information · Legal