loader: Update storage configuration
ClosedPublic
Actions

Authored by anlambert on Feb 21 2020, 3:27 PM.

Details

Reviewers

olasd

Group Reviewers

Reviewers

Commits

rDENVe5446eae09b6: docker/conf/loader: Update storage configuration

Summary

This fixes the loading of origins in the docker environment, see reported error below:

E                     mswh-loader_1                    | Traceback (most recent call last):
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
E                     mswh-loader_1                    |     R = retval = fun(*args, **kwargs)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/scheduler/task.py", line 53, in __call__
E                     mswh-loader_1                    |     result = super().__call__(*args, **kwargs)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/celery/app/trace.py", line 650, in __protected_call__
E                     mswh-loader_1                    |     return self.run(*args, **kwargs)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/git/tasks.py", line 22, in load_git
E                     mswh-loader_1                    |     return loader.load()
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 287, in load
E                     mswh-loader_1                    |     self._store_origin_visit()
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/loader/core/loader.py", line 165, in _store_origin_visit
E                     mswh-loader_1                    |     self.storage.origin_add_one(origin)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 154, in meth_
E                     mswh-loader_1                    |     return self.post(meth._endpoint_path, post_data)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 241, in post
E                     mswh-loader_1                    |     return self._decode_response(response)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 314, in _decode_response
E                     mswh-loader_1                    |     self.raise_for_status(response)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/client.py", line 27, in raise_for_status
E                     mswh-loader_1                    |     super().raise_for_status(response)
E                     mswh-loader_1                    |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/api/__init__.py", line 305, in raise_for_status
E                     mswh-loader_1                    |     raise exception from None
E                     mswh-loader_1                    | swh.core.api.RemoteException: <RemoteException 500 AttributeError: ["'dict' object has no attribute 'url'"]>

Diff Detail

Repository

rDENV Development environment

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Feb 21 2020, 3:27 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 21 2020, 3:27 PM

Harbormaster completed remote builds in B10721: Diff 9654.Feb 21 2020, 3:27 PM

olasd accepted this revision.Feb 21 2020, 3:57 PM

olasd added a subscriber: olasd.

olasd added inline comments.

docker/conf/loader.yml

1–14

could (probably, untested) be shortened to:

storage:
  cls: pipeline
  steps:
  - cls: validate
  - cls: filter
  - cls: buffer
    min_batch_size:
      content: 10000
      content_bytes: 104857600
      directory: 1000
      revision: 1000
  - cls: remote
    url: http://swh-storage:5002/

This revision is now accepted and ready to land.Feb 21 2020, 3:57 PM

Update: Simplify storage configuration

Harbormaster completed remote builds in B10725: Diff 9658.Feb 21 2020, 4:21 PM

anlambert added inline comments.Feb 21 2020, 4:35 PM

docker/conf/loader.yml
1–14	Tested and approved. Too bad that messages coming from the `warnings` module are not redirected to the logging output as I would have spot the syntax improvement otherwise. There is a feature to do it but I do not really see where it should be called when using the docker environment.