Plus that documents it a bit in an automatic manner, so *thumbs up*.

I decided to make the swh-graph container create the compressed graph itself before starting. That's the easiest way to use it AND to implement it IMO.

Issue is now solved, closing this.

awesome, thanks :)

and the build is green ;)

thanks for having investigated that

So the docker_default network did not get removed since April 20, 2021, see docker network inspect docker_default output below:

15:07:16 [
15:07:16     {
15:07:16         "Name": "docker_default",
15:07:16         "Id": "a0145eb2f5b3f055bfafce7a88289346c4668f3a19f2544122267272cd96f7b3",
15:07:16         "Created": "2021-04-20T02:57:51.141593622Z",
15:07:16         "Scope": "local",
15:07:16         "Driver": "bridge",
15:07:16         "EnableIPv6": false,
15:07:16         "IPAM": {
15:07:16             "Driver": "default",
15:07:16             "Options": null,
15:07:16             "Config": [
15:07:16                 {
15:07:16                     "Subnet": "172.28.0.0/16",
15:07:16                     "Gateway": "172.28.0.1"
15:07:16                 }
15:07:16             ]
15:07:16         },
15:07:16         "Internal": false,
15:07:16         "Attachable": true,
15:07:16         "Ingress": false,
15:07:16         "ConfigFrom": {
15:07:16             "Network": ""
15:07:16         },
15:07:16         "ConfigOnly": false,
15:07:16         "Containers": {},
15:07:16         "Options": {},
15:07:16         "Labels": {
15:07:16             "com.docker.compose.network": "default",
15:07:16             "com.docker.compose.project": "docker",
15:07:16             "com.docker.compose.version": "1.29.1"
15:07:16         }
15:07:16     }
15:07:16 ]

If you face this issue, try restarting the containers using docker-compose down and docker-compose up.

So either the vault is fine and the test needs to be improved.
Or the vault builds the tarball without taking into account the symlink case.

@rendong951 Sorry, but we can't reproduce the issue on our end.

I also try command fllowing your reply，the results are the same.

It's not a DNS issue. The error is Could not resolve hostname https:. It shouldn't try to resolve https:, it's a scheme, not a domain name.

Thank you very much for your reply.
I followed your command.[1]

Please wrap your stacktrace, excerpt of code, etc... with triple backquote before and after.
So this becomes more readable.
Thanks in advance.

Thank you very much for your reply.
And I follow your guidance，now，all the services are started，but when I save code in my Software Heritage platform, the status always failed, I checked log :
swh-loader_1 | [2020-12-18 02:49:13,495: ERROR/ForkPoolWorker-1] Loading failure, updating to partial status
swh-loader_1 | Traceback (most recent call last):
swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/dulwich/client.py", line 912, in fetch_pack
swh-loader_1 | refs, server_capabilities = read_pkt_refs(proto)
swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/dulwich/client.py", line 215, in read_pkt_refs
swh-loader_1 | for pkt in proto.read_pkt_seq():
swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/dulwich/protocol.py", line 277, in read_pkt_seq
swh-loader_1 | pkt = self.read_pkt_line()
swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/dulwich/protocol.py", line 223, in read_pkt_line
swh-loader_1 | raise HangupException()
swh-loader_1 | dulwich.errors.HangupException: The remote server unexpectedly closed the connection.
swh-loader_1 |
swh-loader_1 | During handling of the above exception, another exception occurred:
swh-loader_1 |
swh-loader_1 | Traceback (most recent call last):
swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 318, in load
swh-loader_1 | more_data_to_fetch = self.fetch_data()
swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/loader.py", line 239, in fetch_data
swh-loader_1 | self.origin.url, self.base_snapshot, do_progress
swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/loader.py", line 174, in fetch_pack_from_origin
swh-loader_1 | progress=do_activity,
swh-loader_1 | File "/srv/softwareheritage/venv/lib/python3.7/site-packages/dulwich/client.py", line 914, in fetch_pack
swh-loader_1 | raise _remote_error_from_stderr(stderr)
swh-loader_1 | dulwich.errors.HangupException: ssh: Could not resolve hostname https: Temporary failure in name resolution

I have added the `--pull` option in the job configuration so the python image will always be pulled (if needed) during the build :

It seems the python 3.7 images on the jenkins master is quite old :

root@thyssen:~# docker images | grep python
python                     3.7                  a4cc999cf2aa        16 months ago       929MB

I'm guessing it's because $VERSION_CODENAME is empty here: https://forge.softwareheritage.org/source/swh-environment/browse/master/docker/Dockerfile$3

possibly related to T2373.

It worked \m/ (that was it).

Hmph, should have changed the commit title ¯\_(ツ)_/¯

Probably doesn't anymore since we moved to confluent-kafka

Resolved by D2424.

Closed by 9a98e8be06817055693671cfbe6917645171993e

And it's fixed.

Ok, the stacktrace and the current indexer code (master) seems out of sync.
rDCIDXb2ff26454b37f3d332b90ff6a70a45b43dea6526 fixing that part of the journal client.

I failed to find the swh-docker-compose.logs file.
I finally found the reference of the node the build ran on though (because i keep forgetting that) [1]

Indeed, the Dockerfile fix is much simpler. Let's hope they will fix the issue upstream (not sure regarding this comment).

We've migrated away from doing that ages ago in favor of app factories, and I'd very much prefer we avoid introducing this again: Side effects such as reading configuration files unconditionally running in importable modules are very bad form.

Seems the simplest solution is to instantiate the wsgi application in swh.*.api.server modules.

Ah, I guess that's the opposite then.

Works fine for me? Are you sure your docker image is up to date?

I was also looking at it.

Hm, i don't know how to close this issue, but it should be closed now.
Thx.

Thx to this change (provided by @olasd) in the swh-docker-dev repository, tasks are exectued:

diff --git a/conf/loader.yml b/conf/loader.yml
index 4a4fb54..0cc07e6 100644
--- a/conf/loader.yml
+++ b/conf/loader.yml
@@ -5,6 +5,7 @@ storage:
 celery:
   task_broker: amqp://guest:guest@amqp//
   task_modules:
+    - swh.loader.package.tasks
     - swh.loader.debian.tasks
     - swh.loader.dir.tasks
     - swh.loader.git.tasks
@@ -16,6 +17,7 @@ celery:
     - swh.deposit.loader.tasks

Let's call it done, event if the small dataset part has not been addressed.

Let's call it done, some minor parts may still need a bit of attention thou

Closing this as I forgot it exists a configuration entry to force the serving of assets by Django (https://forge.softwareheritage.org/rCDFD7b3213293ca1670a738d540cbba05e87e5cf6042).