Page MenuHomeSoftware Heritage

Add a new unit test image based on swh-tox enriched with kafka
ClosedPublic

Authored by ardumont on Mar 19 2019, 6:23 PM.

Details

Summary

Use case is the journal tests needs a kafka that can be
started/stopped during the tests.

Related D1239

Remains:

  • D1239 needs adaptation to actually lift that, based on environment variable(s)
  • the build job needs to be parametrized to change the jenkins node for running tests. Defaulting to swh-tox and be able to run with swh-kafka (for swh-journal here)
Test Plan
# run a local registry
docker run -d -p 5000:5000 --name registry registry:2
# build the node using local registry
REGISTRY=localhost:5000 ; make kafka
# connect to the node and check everything is as expected
REGISTRY=localhost:5000; make kafka exec /bin/bash
$ ls -lh /opt
lrwxrwxrwx 1 jenkins jenkins   21 Mar 20 10:04 kafka -> /opt/kafka_2.12-2.1.1
drwxr-xr-x 1 jenkins jenkins 4.0K Feb  8 18:33 kafka_2.12-2.1.1

Diff Detail

Repository
rCDFJ Dockerfiles for Jenkins
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Mar 19 2019, 6:23 PM
ardumont marked an inline comment as done.Mar 19 2019, 6:25 PM
ardumont added inline comments.
kafka/retrieve-kafka.sh
32 ↗(On Diff #4016)

That should go away.
Just to demonstrate i tried to have something that was less naive that what ends up in the Dockerfile.

To run this script, i'd need more adaptation to do in the Makefile (mounting volume).
I'm not sure that is worth the trouble though.

olasd requested changes to this revision.Mar 20 2019, 9:32 AM
olasd added a subscriber: olasd.
olasd added inline comments.
kafka/Dockerfile
11–15

please collapse these in a single command, to avoid adding four useless layers in the docker cache

This revision now requires changes to proceed.Mar 20 2019, 9:32 AM
olasd added inline comments.Mar 20 2019, 9:34 AM
kafka/Dockerfile
11–15

you should probably also add the shasum check so we're sure what we've downloaded

vlorentz added inline comments.
kafka/Dockerfile
11–15

and rm the .tgz in the same command.

Thanks people!
will do that.

ardumont updated this revision to Diff 4021.Mar 20 2019, 9:53 AM

Adapt according to reviews

ardumont updated this revision to Diff 4022.Mar 20 2019, 9:57 AM

Remove the signature file as well

olasd accepted this revision.Mar 20 2019, 9:59 AM
This revision is now accepted and ready to land.Mar 20 2019, 9:59 AM
ardumont updated this revision to Diff 4024.Mar 20 2019, 10:29 AM
  • kafka/Dockerfile: Use argument to avoid repetition
  • kafka/Dockerfile: Use multi-stage build
ardumont updated this revision to Diff 4025.Mar 20 2019, 11:07 AM
  • kafka/Dockerfile: Use arguments to avoid repetition
  • kafka/Dockerfile: Use multi-stage build
  • kafka/dockerfile: Fix permission in the runtime image
ardumont edited the test plan for this revision. (Show Details)Mar 20 2019, 11:08 AM
ardumont updated this revision to Diff 4026.Mar 20 2019, 11:46 AM
  • kafka/dockerfile: Name the building stage 0 kafka_fetcher
This revision was automatically updated to reflect the committed changes.