docker tests on Jenkins: error while removing network
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	anlambert
	May 12 2021, 1:46 PM

Description

swh-docker-dev job on Jenkins is failing since April 20, 2021.

The error occurs when calling docker-compose down, network cannot be removed.

03:09:28 Removing docker_swh-objstorage_1               ... done
03:09:28 Removing network docker_default
03:09:28 error while removing network: network docker_default id a0145eb2f5b3f055bfafce7a88289346c4668f3a19f2544122267272cd96f7b3 has active endpoints
03:09:28 ERROR: InvocationError for command '/bin/bash tests/run_tests.sh' (exited with code 1)

This seems specific to the Jenkins environment as I cannot reproduce the error locally using
same docker and docker-compose versions.

Docker version 20.10.5, build 55c4c88
docker-compose version 1.25.0, build unknown

Event Timeline

anlambert triaged this task as Normal priority.May 12 2021, 1:46 PM

anlambert created this task.

So the docker_default network did not get removed since April 20, 2021, see docker network inspect docker_default output below:

15:07:16 [
15:07:16     {
15:07:16         "Name": "docker_default",
15:07:16         "Id": "a0145eb2f5b3f055bfafce7a88289346c4668f3a19f2544122267272cd96f7b3",
15:07:16         "Created": "2021-04-20T02:57:51.141593622Z",
15:07:16         "Scope": "local",
15:07:16         "Driver": "bridge",
15:07:16         "EnableIPv6": false,
15:07:16         "IPAM": {
15:07:16             "Driver": "default",
15:07:16             "Options": null,
15:07:16             "Config": [
15:07:16                 {
15:07:16                     "Subnet": "172.28.0.0/16",
15:07:16                     "Gateway": "172.28.0.1"
15:07:16                 }
15:07:16             ]
15:07:16         },
15:07:16         "Internal": false,
15:07:16         "Attachable": true,
15:07:16         "Ingress": false,
15:07:16         "ConfigFrom": {
15:07:16             "Network": ""
15:07:16         },
15:07:16         "ConfigOnly": false,
15:07:16         "Containers": {},
15:07:16         "Options": {},
15:07:16         "Labels": {
15:07:16             "com.docker.compose.network": "default",
15:07:16             "com.docker.compose.project": "docker",
15:07:16             "com.docker.compose.version": "1.29.1"
15:07:16         }
15:07:16     }
15:07:16 ]

Calling docker network remove docker_default raises the same error, what puzzled me is that no containers seem attached to the network.

thanks for having investigated that

I have restarted the docker daemon and it seems the problem is no longer present. It happens, docker is unfortunately not free of bugs :).

root@thyssen:/var/lib/jenkins/workspace/swh-docker-dev/docker# docker network ls
NETWORK ID     NAME             DRIVER    SCOPE
5a11bd2c4c8d   bridge           bridge    local
a0145eb2f5b3   docker_default   bridge    local
727f6f855ea1   host             host      local
e2969ca4e137   none             null      local
root@thyssen:/var/lib/jenkins/workspace/swh-docker-dev/docker# systemctl restart docker
root@thyssen:/var/lib/jenkins/workspace/swh-docker-dev/docker# docker-compose down -v
Removing network docker_default
root@thyssen:/var/lib/jenkins/workspace/swh-docker-dev/docker# docker network ls
NETWORK ID     NAME      DRIVER    SCOPE
82509cd3269e   bridge    bridge    local
727f6f855ea1   host      host      local
e2969ca4e137   none      null      local

A new build is in progress, let's see if it will succeed

\o/, thanks !

16:50:52 ============================= test session starts ==============================
16:50:52 platform linux -- Python 3.7.3, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
16:50:52 cachedir: .tox/py3/.pytest_cache
16:50:52 rootdir: /var/lib/jenkins/workspace/swh-docker-dev/docker
16:50:52 plugins: testinfra-6.3.0, testinfra-6.0.0
16:50:52 collected 7 items
16:50:52 
16:50:52 tests/test_deposit.py .....                                              [ 71%]
16:53:13 tests/test_git_loader.py .                                               [ 85%]

16:56:04 tests/test_vault.py .                                                    [100%]
16:56:50 
16:56:50 ======================== 7 passed in 358.66s (0:05:58) =========================

and the build is green ;)

19:04:12 Finished: SUCCESS

awesome, thanks :)

Issue is now solved, closing this.

This task has been migrated to GitLab.

docker tests on Jenkins: error while removing networkClosed, MigratedEdits LockedActions

Description

Event Timeline

docker tests on Jenkins: error while removing network
Closed, MigratedEdits Locked
Actions