⚙ D1239 swh-journal: Add tests to the journal publisher using kafka instance

Now i need to concentrate on removing the part about the target ./setup.py develop.
Well not necessarily removing that part (because that could prove useful for others) but at least not make the ci job rely on it (through tox in the current diff's state).

Rebase on latest master

ardumont planned changes to this revision.Mar 19 2019, 10:55 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/42/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/42/console

Harbormaster failed remote builds in B4715: Diff 4000!Mar 19 2019, 10:56 AM

Remove unused import

ardumont planned changes to this revision.Mar 19 2019, 11:41 AM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/43/ for more details.

Harbormaster completed remote builds in B4716: Diff 4001.Mar 19 2019, 11:42 AM

ardumont edited the summary of this revision. (Show Details)Mar 19 2019, 11:48 AM

ardumont mentioned this in D1280: Add a new unit test image based on swh-tox enriched with kafka.Mar 19 2019, 6:23 PM

ardumont retitled this revision from wip: pytest-kafka: ci tryout to swh-journal: Add tests to the journal publisher using kafka instance.Mar 20 2019, 8:55 AM

ardumont edited the summary of this revision. (Show Details)

ardumont added a parent revision: D1280: Add a new unit test image based on swh-tox enriched with kafka.

So, if we keep the setup.py snippet to install kafka, I think it should be a completely separate subcommand of setup.py, rather than overriding the default behavior of the develop subcommand, which has a somewhat standard meaning of "install the current module for a development setup".

But I really think those 50 lines of python should just be turned into 5-10 lines of shell script that would be more readable and do the exact same thing.

(disclaimer: I still haven't looked too closely at the actual tests)

ardumont mentioned this in D1282: Make swh-journal node run on swh-kafka node.Mar 20 2019, 12:39 PM

So, if we keep the setup.py snippet to install kafka, I think it should be a completely separate subcommand of setup.py, rather than overriding the default behavior of the develop subcommand, which has a somewhat standard meaning of "install the current module for a development setup".

well, it was for that indeed, for the tests to work so for development.

(I don't know yet how to make a subcommand of setup.py, i just started from the pytest-kafka's intial setup.py which does that)

But I really think those 50 lines of python should just be turned into 5-10 lines of shell script that would be more readable and do the exact same thing.

fine by me, i had it in D1280 (prior to removing it) so i'll just take it back for here.

(disclaimer: I still haven't looked too closely at the actual tests)

ack, that's fine ;)

For information, it's just:

write to the publisher (as our swh.storage.listener does)
publisher reacts, reifies objects and publish them
a consumer of the publisher asserts the objects are the expected reified ones

README: Explain how to run tests
setup.py: Use a shell script to ease the kafka development setup
Makefile.local: Add a target to clean up cache artifacts

This will break test as the tox has been adapted to remove the
dependency on setup.py develop. This will be addressed when D1280,
D1282 converge and land.

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/44/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/44/console

Harbormaster failed remote builds in B4755: Diff 4033!Mar 20 2019, 2:08 PM

ardumont edited the summary of this revision. (Show Details)Mar 20 2019, 2:11 PM

ardumont added a parent revision: D1282: Make swh-journal node run on swh-kafka node.

bin/install-kafka.sh: Fix indentation

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/45/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/45/console

Harbormaster failed remote builds in B4756: Diff 4034!Mar 20 2019, 2:22 PM

bin/install-kafka.sh: Fix indentation for real this time

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/46/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/46/console

Harbormaster failed remote builds in B4757: Diff 4035!Mar 20 2019, 2:24 PM

bin/install-kafka.sh: Fix indentation

After fixing my emacs setup:

;; default indentation setup
(setq-default indent-tabs-mode nil
              tab-width 4)

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/47/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/47/console

Harbormaster failed remote builds in B4758: Diff 4036!Mar 20 2019, 2:33 PM

ardumont mentioned this in rCDFJcc483aa7eb7e: Add a new unit test image based on swh-tox enriched with kafka.Mar 20 2019, 5:44 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/48/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/48/console

ardumont mentioned this in rCDFJ69e51d5f9671: kafka/Dockerfile: Expose SWH_KAFKA_ROOT.Mar 21 2019, 11:20 AM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/49/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/49/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/50/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/50/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/51/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/51/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/52/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/52/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/53/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/53/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/54/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/54/console

tests/conftest: Ensure failure message if inexisting kafka install
tox: Pass specific environment variable to tox
setup.py: Get back to initial formatting

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/55/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/55/console

Harbormaster failed remote builds in B4769: Diff 4050!Mar 22 2019, 3:50 PM

Fix pep8 violation

installed back the git-pre-commit hook to avoid that again

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/56/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/56/console

Harbormaster failed remote builds in B4770: Diff 4051!Mar 22 2019, 3:55 PM

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/57/ for more details.

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/58/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/58/console

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/59/ for more details.

Harbormaster completed remote builds in B4770: Diff 4051.Mar 22 2019, 4:05 PM

bin/install-kafka.sh: Regroup variables together

Following oral discussion with @douardda

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/60/ for more details.

Harbormaster completed remote builds in B4773: Diff 4054.Mar 22 2019, 4:13 PM

test_publisher_kafka: Rename test_publisher to make a nicer/smaller diff

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/61/ for more details.

Harbormaster completed remote builds in B4774: Diff 4055.Mar 22 2019, 4:16 PM

Rework commit messages

Aside from that, nothing changed

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DJNL/job/tox/62/
See console output for more information: https://jenkins.softwareheritage.org/job/DJNL/job/tox/62/console

Harbormaster failed remote builds in B4784: Diff 4075!Mar 25 2019, 11:35 AM

tests: Inline the kafka_consumer pytest fixture
tests: Simplify initialization test to use fixtures consistently

Build is green
See https://jenkins.softwareheritage.org/job/DJNL/job/tox/63/ for more details.

Harbormaster completed remote builds in B4791: Diff 4081.Mar 25 2019, 3:51 PM

This revision was not accepted when it landed; it landed in state Needs Review.Mar 25 2019, 3:53 PM

Closed by commit rDJNLfa53bac231c3: Bootstrap publisher tests with kafka instance (authored by ardumont). · Explain Why

This revision was automatically updated to reflect the committed changes.

ardumont mentioned this in rDENVc5a48242d23b: journal-publisher: Update the journal publisher entrypoint.Nov 26 2019, 1:35 PM

swh-journal: Add tests to the journal publisher using kafka instance
ClosedPublic
Actions

Details

Diff Detail

Event Timeline

Revision Contents
Changeset List

Diff 4081

.gitignore

Makefile.local

README.md

bin/install-kafka.sh

requirements-test.txt

swh/journal/cli.py

swh/journal/publisher.py

swh/journal/tests/conftest.py

swh/journal/tests/test_publisher.py

swh/journal/tests/test_publisher_kafka.py

swh/journal/tests/test_publisher_no_kafka.py

tox.ini

swh-journal: Add tests to the journal publisher using kafka instanceClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 4081

.gitignore

Makefile.local

README.md

bin/install-kafka.sh

requirements-test.txt

swh/journal/cli.py

swh/journal/publisher.py

swh/journal/tests/conftest.py

swh/journal/tests/test_publisher.py

swh/journal/tests/test_publisher_kafka.py

swh/journal/tests/test_publisher_no_kafka.py

tox.ini

swh-journal: Add tests to the journal publisher using kafka instance
ClosedPublic
Actions

Revision Contents
Changeset List