Page MenuHomeSoftware Heritage

Add a new unit test image based on swh-tox enriched with kafka
ClosedPublic

Authored by ardumont on Mar 19 2019, 6:23 PM.

Details

Summary

Use case is the journal tests needs a kafka that can be
started/stopped during the tests.

Related D1239

Remains:

  • D1239 needs adaptation to actually lift that, based on environment variable(s)
  • the build job needs to be parametrized to change the jenkins node for running tests. Defaulting to swh-tox and be able to run with swh-kafka (for swh-journal here)
Test Plan
# run a local registry
docker run -d -p 5000:5000 --name registry registry:2
# build the node using local registry
REGISTRY=localhost:5000 ; make kafka
# connect to the node and check everything is as expected
REGISTRY=localhost:5000; make kafka exec /bin/bash
$ ls -lh /opt
lrwxrwxrwx 1 jenkins jenkins   21 Mar 20 10:04 kafka -> /opt/kafka_2.12-2.1.1
drwxr-xr-x 1 jenkins jenkins 4.0K Feb  8 18:33 kafka_2.12-2.1.1

Diff Detail

Repository
rCDFJ Dockerfiles for Jenkins
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont added inline comments.
kafka/retrieve-kafka.sh
32 ↗(On Diff #4016)

That should go away.
Just to demonstrate i tried to have something that was less naive that what ends up in the Dockerfile.

To run this script, i'd need more adaptation to do in the Makefile (mounting volume).
I'm not sure that is worth the trouble though.

olasd requested changes to this revision.Mar 20 2019, 9:32 AM
olasd added a subscriber: olasd.
olasd added inline comments.
kafka/Dockerfile
10–14

please collapse these in a single command, to avoid adding four useless layers in the docker cache

This revision now requires changes to proceed.Mar 20 2019, 9:32 AM
kafka/Dockerfile
10–14

you should probably also add the shasum check so we're sure what we've downloaded

vlorentz added inline comments.
kafka/Dockerfile
10–14

and rm the .tgz in the same command.

Adapt according to reviews

Remove the signature file as well

This revision is now accepted and ready to land.Mar 20 2019, 9:59 AM
  • kafka/Dockerfile: Use argument to avoid repetition
  • kafka/Dockerfile: Use multi-stage build
  • kafka/Dockerfile: Use arguments to avoid repetition
  • kafka/Dockerfile: Use multi-stage build
  • kafka/dockerfile: Fix permission in the runtime image
  • kafka/dockerfile: Name the building stage 0 kafka_fetcher
This revision was automatically updated to reflect the committed changes.