Add a job running swh-mirror tests
ClosedPublic
Actions

Authored by lunar on Oct 13 2022, 11:13 AM.

Details

Reviewers

olasd

Group Reviewers

System administrators

Commits

rCJSWH3f35eb074647: Add a job running swh-mirror tests

Summary

swh-mirror tests are quite involved as they deploy a replicated Software
Heritage Archive in the form of a Docker swarm.

This new swh-mirror/run-tests job assumes that Jenkins running has
access to a Docker engine. It needs:

to be able to build images;
to be configured as part of a swarm;
to have at least one node with the following labels:

org.softwareheritage.mirror.monitoring=true
org.softwareheritage.mirror.volumes.storage-db=true
org.softwareheritage.mirror.volumes.web-db=true
org.softwareheritage.mirror.volumes.objstorage=true
org.softwareheritage.mirror.volumes.redis=true

The pipeline will first create up-to-date Docker images used by the
swarm (using Jenkins host) and then run pytest in a Docker container
using swh-jenkins/tox.

As it depends on network resources, the test can drag for quite long.
The timeout is currently set at one hour. In most cases the pipeline
will be done in 15-20 minutes.

Care is taken both in the test code and within the Jenkins pipeline to
remove resources related to the Docker swarm (stack, containers, configs
and volumes) on teardown.

The job will need the changes proposed as D8634 to work.

Diff Detail

Repository

rCJSWH Jenkins jobs

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

lunar created this revision.Oct 13 2022, 11:13 AM

Herald added a reviewer: System administrators. · View Herald TranscriptOct 13 2022, 11:13 AM

lunar requested review of this revision.Oct 13 2022, 11:13 AM

Harbormaster completed remote builds in B32271: Diff 31320.Oct 13 2022, 11:13 AM

Here is screenshot of the resource monitor on my system while running the test:

anlambert added a subscriber: anlambert.Oct 13 2022, 11:26 AM

anlambert added inline comments.

jobs/templates/swh-mirror-run-tests.groovy.j2
42–52	Found the issue, as explained by the raised exception when running Jenkins jib builder: `jinja2.exceptions.TemplateNotFound: includes/agent-docker.groovy.j2`, the template file could not be found. Jenkins job builder includes files relative to the yaml file describing the job so the correct way to include the docker agent here is the following:* {% filter indent(width=8) %} {%- include 'templates/includes/agent-docker.groovy.j2' -%} {% endfilter %}