Page MenuHomeSoftware Heritage
Differential D4661

search.cli: Subscribe journal client to origin_intrinsic_metadata topic
ClosedPublic
Actions

Authored by ardumont on Dec 4 2020, 10:37 AM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2590: Finish the indexer -> swh-search pipeline
Commits
rDSEA27f82bafeaa4: cli: Subscribe journal client to origin_intrinsic_metadata topic
Summary

Depends on D4671

Test Plan

tox
(failing cause the master one is already failing [1] [2])

[1] https://jenkins.softwareheritage.org/job/DSEA/job/tests/480/

[2] T2791

Diff Detail

Repository
rDSEA Archive search
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Dec 4 2020, 10:37 AM
swh-public-ci added a comment.Dec 4 2020, 10:38 AM

Build has FAILED

Patch application report for D4661 (id=16541)

Rebasing onto c3e5bf0a11...

Current branch diff-target is up to date.
Changes applied before test
commit 7524903bb8f9dfb8ebffdc13a3dc1e9ccbc91940
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 10:36:02 2020 +0100

    search.cli: Subscribe journal client to origin_intrinsic_metadata topic

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/41/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/41/console

Harbormaster failed remote builds in B17727: Diff 16541!Dec 4 2020, 10:38 AM
ardumont edited the test plan for this revision. (Show Details)Dec 4 2020, 10:59 AM
ardumont edited the test plan for this revision. (Show Details)Dec 4 2020, 11:02 AM
vlorentz accepted this revision.Dec 4 2020, 11:25 AM
This revision is now accepted and ready to land.Dec 4 2020, 11:25 AM
vlorentz added a task: T2590: Finish the indexer -> swh-search pipeline.Dec 4 2020, 1:41 PM
olasd added a subscriber: olasd.Dec 4 2020, 4:46 PM
olasd added inline comments.
swh/search/cli.py
75 ↗(On Diff #16541)

I guess this should be updated for origin_visit_statuses.

ardumont added a comment.Dec 4 2020, 5:32 PM

wondering now if that's actually ok

We are using the prefix swh.journal.objects.* for our model objects but swh.journal.indexed.* o

swh/search/cli.py
75 ↗(On Diff #16541)

indeed, i guess we could as well.

ardumont added inline comments.Dec 4 2020, 5:34 PM
swh/search/cli.py
75 ↗(On Diff #16541)

Also, come to think of it, the actual origin-intrinsic-metadata won't work.
They are using different prefixed topics:

  • swh.journal.objects.{origin,origin_visit,origin_visit_status}
  • swh.journal.indexed.origin_intrinsic_metadata

That means, we will have to improve that cli to allow the user to define the prefix and the object types to use (or use the configuration for that).

Either way, In other diffs ^

ardumont updated this revision to Diff 16574.Dec 7 2020, 8:55 AM

Rebase on top of D4671

ardumont edited the summary of this revision. (Show Details)Dec 7 2020, 8:55 AM
ardumont added a parent revision: D4671: cli: Subscribe journal client to origin_visit_status.
swh-public-ci added a comment.Dec 7 2020, 8:58 AM

Build has FAILED

Patch application report for D4661 (id=16574)

Could not rebase; Attempt merge onto c3e5bf0a11...

Updating c3e5bf0..bb49a06
Fast-forward
 swh/search/cli.py            |  26 ++++++--
 swh/search/journal_client.py |  15 +++++
 swh/search/tests/test_cli.py | 140 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 176 insertions(+), 5 deletions(-)
Changes applied before test
commit bb49a0630e2ee19925d001dbd10aba66481c90e6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 10:36:02 2020 +0100

    cli: Subscribe journal client to origin_intrinsic_metadata topic
    
    Related to T2590

commit 9646009720da7d2110d4dd07c62ba4795b0ecba6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:26:50 2020 +0100

    cli: Subscribe journal client to origin_visit_status
    
    Related to T2590

commit 408d51ef571391b85edf4ab570b1280ff5761aea
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:39:21 2020 +0100

    cli: Allow topic prefix declaration through cli or configuration
    
    This will allow using multiple instances of journal client both for subscribing
    to different prefixed topics (swh.journal.objects and swh.journal.indexed).
    
    Related to T2590

commit 8ffcec124b73ecc2d5362cdc0429902ec6c2be27
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:34:41 2020 +0100

    cli: Allow object-type declaration through cli or configuration
    
    This will allow using multiple instances of journal client both for subscribing
    to different topics.
    
    Related to T2590

Link to build: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/45/
See console output for more information: https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/45/console

Harbormaster failed remote builds in B17756: Diff 16574!Dec 7 2020, 8:58 AM
ardumont added inline comments.Dec 7 2020, 8:58 AM
swh/search/cli.py
75 ↗(On Diff #16541)

I guess this should be updated for origin_visit_statuses.

indeed, i guess we could as well.

D4671

That means, we will have to improve that cli to allow the user to define the
prefix and the object types to use (or use the configuration for that).
Either way, In other diffs ^

Respectively D4670 D4669

swh-public-ci added a comment.Dec 7 2020, 10:05 PM

Build is green

Patch application report for D4661 (id=16574)

Could not rebase; Attempt merge onto c3e5bf0a11...

Updating c3e5bf0..bb49a06
Fast-forward
 swh/search/cli.py            |  26 ++++++--
 swh/search/journal_client.py |  15 +++++
 swh/search/tests/test_cli.py | 140 ++++++++++++++++++++++++++++++++++++++++++-
 3 files changed, 176 insertions(+), 5 deletions(-)
Changes applied before test
commit bb49a0630e2ee19925d001dbd10aba66481c90e6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 10:36:02 2020 +0100

    cli: Subscribe journal client to origin_intrinsic_metadata topic
    
    Related to T2590

commit 9646009720da7d2110d4dd07c62ba4795b0ecba6
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:26:50 2020 +0100

    cli: Subscribe journal client to origin_visit_status
    
    Related to T2590

commit 408d51ef571391b85edf4ab570b1280ff5761aea
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:39:21 2020 +0100

    cli: Allow topic prefix declaration through cli or configuration
    
    This will allow using multiple instances of journal client both for subscribing
    to different prefixed topics (swh.journal.objects and swh.journal.indexed).
    
    Related to T2590

commit 8ffcec124b73ecc2d5362cdc0429902ec6c2be27
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:34:41 2020 +0100

    cli: Allow object-type declaration through cli or configuration
    
    This will allow using multiple instances of journal client both for subscribing
    to different topics.
    
    Related to T2590

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/46/ for more details.

Harbormaster completed remote builds in B17756: Diff 16574.Dec 7 2020, 10:05 PM
ardumont updated this revision to Diff 16603.Dec 8 2020, 9:51 AM

Rebase

swh-public-ci added a comment.Dec 8 2020, 9:53 AM

Build is green

Patch application report for D4661 (id=16603)

Could not rebase; Attempt merge onto 408d51ef57...

Updating 408d51e..27f82ba
Fast-forward
 swh/search/cli.py            |   2 +-
 swh/search/journal_client.py |  15 ++++++
 swh/search/tests/test_cli.py | 122 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 138 insertions(+), 1 deletion(-)
Changes applied before test
commit 27f82bafeaa4666c9b2a2eba69a73810f3566d68
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 10:36:02 2020 +0100

    cli: Subscribe journal client to origin_intrinsic_metadata topic
    
    Related to T2590

commit 94d305509e7b5e63511a3f636440c74cb4419602
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri Dec 4 21:26:50 2020 +0100

    cli: Subscribe journal client to origin_visit_status
    
    Related to T2590

See https://jenkins.softwareheritage.org/job/DSEA/job/tests-on-diff/51/ for more details.

Harbormaster completed remote builds in B17782: Diff 16603.Dec 8 2020, 9:53 AM
Closed by commit rDSEA27f82bafeaa4: cli: Subscribe journal client to origin_intrinsic_metadata topic (authored by ardumont). · Explain WhyDec 8 2020, 9:59 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSEA27f82bafeaa4: cli: Subscribe journal client to origin_intrinsic_metadata topic.

Revision Contents
Changeset List

PathSize
swh/
search/
tests/
70 lines

Diff 16605

View Options

swh/search/tests/test_cli.py

Loading...
About · Developer information · Legal