Page MenuHomeSoftware Heritage

docker/conf/loader: Update storage configuration
ClosedPublic

Authored by anlambert on Feb 21 2020, 3:27 PM.

Details

Summary

This fixes the loading of origins in the docker environment, see reported error below:

E                     mswh-loader_1                    | Traceback (most recent call last):
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
E                     mswh-loader_1                    |     R = retval = fun(*args, **kwargs)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 53, in __call__
E                     mswh-loader_1                    |     result = super().__call__(*args, **kwargs)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 650, in __protected_call__
E                     mswh-loader_1                    |     return self.run(*args, **kwargs)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/tasks.py", line 22, in load_git
E                     mswh-loader_1                    |     return loader.load()
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 287, in load
E                     mswh-loader_1                    |     self._store_origin_visit()
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 165, in _store_origin_visit
E                     mswh-loader_1                    |     self.storage.origin_add_one(origin)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 154, in meth_
E                     mswh-loader_1                    |     return self.post(meth._endpoint_path, post_data)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 241, in post
E                     mswh-loader_1                    |     return self._decode_response(response)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 314, in _decode_response
E                     mswh-loader_1                    |     self.raise_for_status(response)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 27, in raise_for_status
E                     mswh-loader_1                    |     super().raise_for_status(response)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 305, in raise_for_status
E                     mswh-loader_1                    |     raise exception from None
E                     mswh-loader_1                    | swh.core.api.RemoteException: <RemoteException 500 AttributeError: ["'dict' object has no attribute 'url'"]>

Diff Detail

Repository
rDENV Development environment
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

olasd added a subscriber: olasd.
olasd added inline comments.
docker/conf/loader.yml
1–14

could (probably, untested) be shortened to:

storage:
  cls: pipeline
  steps:
  - cls: validate
  - cls: filter
  - cls: buffer
    min_batch_size:
      content: 10000
      content_bytes: 104857600
      directory: 1000
      revision: 1000
  - cls: remote
    url: http://swh-storage:5002/
This revision is now accepted and ready to land.Feb 21 2020, 3:57 PM

Update: Simplify storage configuration

docker/conf/loader.yml
1–14

Tested and approved.

Too bad that messages coming from the warnings module are not redirected to the logging output as I would have spot the syntax improvement otherwise. There is a feature to do it but I do not really see where it should be called when using the docker environment.