[docker-compose] conf/objstorage: Use a higher value for client_max_size
ClosedPublic
Actions

Authored by anlambert on Mar 6 2019, 4:32 PM.

Details

Reviewers

douardda

Group Reviewers

Reviewers

Commits

rDENV9f8ea5e24591: conf/objstorage: Use a higher value for client_max_size
rCDFD9f8ea5e24591: conf/objstorage: Use a higher value for client_max_size

Summary

When trying to load a git repository from the docker-compose environment, I observed
the following type of errors:

swh-loader_1                  | Traceback (most recent call last):
swh-loader_1                  |   File "/home/swh/.local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 895, in load
swh-loader_1                  |     self.store_data()
swh-loader_1                  |   File "/home/swh/.local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 1002, in store_data
swh-loader_1                  |     self.send_batch_contents(self.get_contents())
swh-loader_1                  |   File "/home/swh/.local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 648, in send_batch_contents
swh-loader_1                  |     packet_size_bytes=packet_size_bytes)
swh-loader_1                  |   File "/home/swh/.local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 41, in send_in_packets
swh-loader_1                  |     sender(formatted_objects)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/retrying.py", line 49, in wrapped_f
swh-loader_1                  |     return Retrying(*dargs, **dkw).call(f, *args, **kw)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/retrying.py", line 206, in call
swh-loader_1                  |     return attempt.get(self._wrap_exception)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/retrying.py", line 247, in get
swh-loader_1                  |     six.reraise(self.value[0], self.value[1], self.value[2])
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/six.py", line 693, in reraise
swh-loader_1                  |     raise value
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/retrying.py", line 200, in call
swh-loader_1                  |     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
swh-loader_1                  |   File "/home/swh/.local/lib/python3.6/site-packages/swh/loader/core/loader.py", line 401, in send_contents
swh-loader_1                  |     self.storage.content_add(content_list)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/storage/api/client.py", line 23, in content_add
swh-loader_1                  |     return self.post('content/add', {'content': content})
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/core/api/__init__.py", line 185, in post
swh-loader_1                  |     return self._decode_response(response)
swh-loader_1                  |   File "/usr/local/lib/python3.6/site-packages/swh/core/api/__init__.py", line 220, in _decode_response
swh-loader_1                  |     raise pickle.loads(decode_response(response))
swh-loader_1                  | swh.core.api.RemoteException: Unexpected status code for API request: 413 (b'Maximum request body size 104857600 exceeded, actual body size 104887327')

This is due to the objstorage client_max_size configuration entry whose value
is too low. So reinstate the value that was always used until the recent configuration
management changes.

Diff Detail

Repository

rCDFD Dockerfiles for developers

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Mar 6 2019, 4:32 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMar 6 2019, 4:32 PM

Harbormaster completed remote builds in B4551: Diff 3840.Mar 6 2019, 4:32 PM

anlambert edited the summary of this revision. (Show Details)Mar 6 2019, 9:26 PM

anlambert retitled this revision from dokcer-compose/conf/objstorage: Use a higher value for client_max_size to [docker-compose] /conf/objstorage: Use a higher value for client_max_size.Mar 7 2019, 11:49 AM

anlambert retitled this revision from [docker-compose] /conf/objstorage: Use a higher value for client_max_size to [docker-compose] conf/objstorage: Use a higher value for client_max_size.

I'm ok with this, but whichever client_max_size we set, we might encounter this traceback. The question is then: how should we handle this the proper way? But it's another story,

This revision is now accepted and ready to land.Mar 8 2019, 10:37 AM

Closed by commit rCDFD9f8ea5e24591: conf/objstorage: Use a higher value for client_max_size (authored by anlambert). · Explain WhyMar 8 2019, 11:01 AM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDENV9f8ea5e24591: conf/objstorage: Use a higher value for client_max_size.Nov 26 2019, 1:35 PM