Page MenuHomeSoftware Heritage
Differential D1326

Add DirectKafkaWriter, which collaborates with the Storage to write to Kafka.
ClosedPublic
Actions

Authored by vlorentz on Apr 1 2019, 3:05 PM.

Details

Reviewers
ardumont
olasd
Group Reviewers
Reviewers
Commits
rDJNL91588e74ae6f: Add DirectKafkaWriter, which collaborates with the Storage to write to Kafka.

Diff Detail

Repository
rDJNL Journal infrastructure
Branch
add-DirectKafkaWriter
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 5048
Build 6784: tox-on-jenkinsJenkins
Build 6783: arc lint + arc unit

Event Timeline

vlorentz created this revision.Apr 1 2019, 3:05 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 1 2019, 3:05 PM
vlorentz updated this revision to Diff 4218.Apr 1 2019, 3:06 PM
  • Remove debug print.
vlorentz added reviewers: ardumont, olasd.Apr 1 2019, 3:06 PM
swh-public-ci added a comment.Apr 1 2019, 3:07 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/73/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/73/console

Harbormaster failed remote builds in B4983: Diff 4217!Apr 1 2019, 3:08 PM
swh-public-ci added a comment.Apr 1 2019, 3:10 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/74/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/74/console

Harbormaster failed remote builds in B4984: Diff 4218!Apr 1 2019, 3:10 PM
swh-public-ci added a comment.Apr 1 2019, 5:58 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/76/ for more details.

ardumont requested changes to this revision.Apr 2 2019, 12:43 AM

I don't remember what's for. As it's not explained in the diff, i might be offtrack here.

Why don't you open a cli endpoint to make that runnable from there?

Requested change to at least have a docstring explaining this.

Thanks.

swh/journal/direct_writer.py
15

Adding docstring to explain the goal would be nice.

40

for my information, what's the meaning of the _ at the end?

I guess the _ as prefix is for private scope meaning, feel free to correct me ;)

This revision now requires changes to proceed.Apr 2 2019, 12:43 AM
vlorentz updated this revision to Diff 4266.Apr 3 2019, 11:45 AM
vlorentz marked an inline comment as done.
  • Add docstring to DirectKafkaWriter.
vlorentz added inline comments.Apr 3 2019, 11:50 AM
swh/journal/direct_writer.py
40

To avoid name shadowing. This is inspired by the PEP8 recommendation for argument names:

If a function argument's name clashes with a reserved keyword, it is generally better to append a single trailing underscore rather than use an abbreviation or spelling corruption. Thus class_ is better than clss. (Perhaps better is to avoid such clashes by using a synonym.)

swh-public-ci added a comment.Apr 3 2019, 11:50 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/84/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/84/console

Harbormaster failed remote builds in B5033: Diff 4266!Apr 3 2019, 11:50 AM
ardumont accepted this revision.Apr 3 2019, 2:13 PM

Sounds fine.

Maybe wait and adapt for D1329 ;)

swh/journal/direct_writer.py
40

Thanks!

swh/journal/tests/test_direct_writer.py
81

That will break with your change in D1329.

This revision is now accepted and ready to land.Apr 3 2019, 2:13 PM
vlorentz updated this revision to Diff 4281.Apr 3 2019, 3:38 PM
  • Rebase + add support for origin_visit.
vlorentz added a parent revision: D1336: Don't leak origin ids to the journal..Apr 3 2019, 3:39 PM
swh-public-ci added a comment.Apr 3 2019, 3:41 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/85/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/85/console

Harbormaster failed remote builds in B5048: Diff 4281!Apr 3 2019, 3:41 PM
vlorentz added a child revision: D1340: Reuse the same Kafka instance across tests..Apr 3 2019, 4:05 PM
ardumont added inline comments.Apr 3 2019, 6:09 PM
swh/journal/tests/test_direct_writer.py
81

snapshot?

vlorentz updated this revision to Diff 4294.Apr 4 2019, 11:52 AM

Add missing object_type.

vlorentz marked 4 inline comments as done.Apr 4 2019, 11:55 AM
swh-public-ci added a comment.Apr 4 2019, 11:56 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/88/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/88/console

Harbormaster failed remote builds in B5096: Diff 4294!Apr 4 2019, 11:56 AM
vlorentz updated this revision to Diff 4295.Apr 4 2019, 11:58 AM

Actually add missing object_type.

swh-public-ci added a comment.Apr 4 2019, 12:01 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/89/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/89/console

Harbormaster failed remote builds in B5097: Diff 4295!Apr 4 2019, 12:01 PM
This revision was landed with ongoing or failed builds.Apr 4 2019, 12:25 PM
Closed by commit rDJNL91588e74ae6f: Add DirectKafkaWriter, which collaborates with the Storage to write to Kafka. (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

PathSize
swh/
journal/
64 lines
6 lines
tests/
10 lines
95 lines
6 lines

Diff 4281

View Options

swh/journal/direct_writer.py

Loading...
View Options

swh/journal/replay.py

Loading...
View Options

swh/journal/tests/conftest.py

Loading...
View Options

swh/journal/tests/test_direct_writer.py

Loading...
View Options

swh/journal/tests/test_replay.py

Loading...
About · Developer information · Legal