cli: Allow topic prefix declaration through cli or configuration
ClosedPublic
Actions

Authored by ardumont on Dec 7 2020, 8:53 AM.

Details

Reviewers

douardda

Group Reviewers

Reviewers

Maniphest Tasks

T2590: Finish the indexer -> swh-search pipeline

Commits

rDSEA408d51ef5713: cli: Allow topic prefix declaration through cli or configuration

Summary

This will allow using multiple instances of journal client both for subscribing
to different prefixed topics (swh.journal.objects and swh.journal.indexed).

Related to T2590
Depends on D4669

Test Plan

tox
(might be failing for unrelated reasons)

Diff Detail

Repository

rDSEA Archive search

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Dec 7 2020, 8:53 AM

ardumont edited the summary of this revision. (Show Details)Dec 7 2020, 8:53 AM

ardumont added a parent revision: D4669: cli: Allow object-type declaration through cli or configuration.

ardumont added a child revision: D4671: cli: Subscribe journal client to origin_visit_status.Dec 7 2020, 8:54 AM

Build has FAILED

Patch application report for D4670 (id=16572)

Could not rebase; Attempt merge onto c3e5bf0a11...

Updating c3e5bf0..408d51e
Fast-forward
 swh/search/cli.py            | 26 ++++++++++++++++++++++----
 swh/search/tests/test_cli.py | 20 ++++++++++++++++++--
 2 files changed, 40 insertions(+), 6 deletions(-)

Changes applied before test

commit 408d51ef571391b85edf4ab570b1280ff5761aea
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:39:21 2020 +0100

    cli: Allow topic prefix declaration through cli or configuration
    
    This will allow using multiple instances of journal client both for subscribing
    to different prefixed topics (swh.journal.objects and swh.journal.indexed).
    
    Related to T2590

commit 8ffcec124b73ecc2d5362cdc0429902ec6c2be27
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:34:41 2020 +0100

    cli: Allow object-type declaration through cli or configuration
    
    This will allow using multiple instances of journal client both for subscribing
    to different topics.
    
    Related to T2590

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/43/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/43/console

Harbormaster failed remote builds in B17754: Diff 16572!Dec 7 2020, 8:55 AM

ardumont mentioned this in D4661: search.cli: Subscribe journal client to origin_intrinsic_metadata topic.Dec 7 2020, 8:58 AM

not sure why this is needed but otherwise lgtm

This revision is now accepted and ready to land.Dec 7 2020, 4:06 PM

@douardda There will be 2 differents prefixed topics:

one running ... --object-type origin --object-type origin_visit --object-type origin_visit_status --prefix swh.journal.objects (or through configuration ymmv).
another running ... --object-type origin_intrinsic_metadata --prefix swh.journal.indexed

without this diff and the previous one, we can't do that ^

Build is green

Patch application report for D4670 (id=16572)

Could not rebase; Attempt merge onto c3e5bf0a11...

Updating c3e5bf0..408d51e
Fast-forward
 swh/search/cli.py            | 26 ++++++++++++++++++++++----
 swh/search/tests/test_cli.py | 20 ++++++++++++++++++--
 2 files changed, 40 insertions(+), 6 deletions(-)

Changes applied before test

commit 408d51ef571391b85edf4ab570b1280ff5761aea
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:39:21 2020 +0100

    cli: Allow topic prefix declaration through cli or configuration
    
    This will allow using multiple instances of journal client both for subscribing
    to different prefixed topics (swh.journal.objects and swh.journal.indexed).
    
    Related to T2590

commit 8ffcec124b73ecc2d5362cdc0429902ec6c2be27
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:34:41 2020 +0100

    cli: Allow object-type declaration through cli or configuration
    
    This will allow using multiple instances of journal client both for subscribing
    to different topics.
    
    Related to T2590

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/48/ for more details.

Harbormaster completed remote builds in B17754: Diff 16572.Dec 7 2020, 10:09 PM

Closed by commit rDSEA408d51ef5713: cli: Allow topic prefix declaration through cli or configuration (authored by ardumont). · Explain WhyDec 8 2020, 9:48 AM

This revision was automatically updated to reflect the committed changes.

ardumont added a commit: rDSEA408d51ef5713: cli: Allow topic prefix declaration through cli or configuration.