Page MenuHomeSoftware Heritage

Add a job running swh-mirror tests
ClosedPublic

Authored by lunar on Oct 13 2022, 11:13 AM.

Details

Summary

swh-mirror tests are quite involved as they deploy a replicated Software
Heritage Archive in the form of a Docker swarm.

This new swh-mirror/run-tests job assumes that Jenkins running has
access to a Docker engine. It needs:

  1. to be able to build images;
  2. to be configured as part of a swarm;
  3. to have at least one node with the following labels:
org.softwareheritage.mirror.monitoring=true
org.softwareheritage.mirror.volumes.storage-db=true
org.softwareheritage.mirror.volumes.web-db=true
org.softwareheritage.mirror.volumes.objstorage=true
org.softwareheritage.mirror.volumes.redis=true

The pipeline will first create up-to-date Docker images used by the
swarm (using Jenkins host) and then run pytest in a Docker container
using swh-jenkins/tox.

As it depends on network resources, the test can drag for quite long.
The timeout is currently set at one hour. In most cases the pipeline
will be done in 15-20 minutes.

Care is taken both in the test code and within the Jenkins pipeline to
remove resources related to the Docker swarm (stack, containers, configs
and volumes) on teardown.

The job will need the changes proposed as D8634 to work.

Diff Detail

Repository
rCJSWH Jenkins jobs
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32735
Build 51287: arc lint + arc unit

Event Timeline

Here is screenshot of the resource monitor on my system while running the test:

anlambert added inline comments.
jobs/templates/swh-mirror-run-tests.groovy.j2
42–52

Found the issue, as explained by the raised exception when running Jenkins jib builder: jinja2.exceptions.TemplateNotFound: includes/agent-docker.groovy.j2, the template file could not be found. Jenkins job builder includes files relative to the yaml file describing the job so the correct way to include the docker agent here is the following:*

{% filter indent(width=8) %}
  {%- include 'templates/includes/agent-docker.groovy.j2' -%}
{% endfilter %}

Use the template appropriately to run the Docker agent

lunar added inline comments.
jobs/templates/swh-mirror-run-tests.groovy.j2
42–52

I should have thought about it but this was really not obvious. Thank you!

Screenshot of resources usage while test runs on my laptop:

This revision is now accepted and ready to land.Nov 7 2022, 1:41 PM
This revision was automatically updated to reflect the committed changes.