Page MenuHomeSoftware Heritage

api.server: Fix client_max_size parameter transmission to aiohttp
ClosedPublic

Authored by anlambert on Apr 26 2019, 4:15 PM.

Details

Summary

Since a recent refactoring, the client_max_size config value was no more
transmitted to the aiohttp.web.Application instance.

This was resulting in errors when a cooked archive had a size greater than 1048576 bytes,
see stack trace observed when playing with the vault in docker-compose environment below:

swh-vault-worker_1            | [2019-04-26 13:05:13,855: INFO/MainProcess] Connected to amqp://guest:**@amqp:5672//
swh-vault-worker_1            | [2019-04-26 13:05:13,988: INFO/MainProcess] vault@2713bd8ecd80 ready.
swh-vault-worker_1            | [2019-04-26 13:11:42,416: INFO/MainProcess] Received task: swh.vault.cooking_tasks.SWHCookingTask[2cf11e3e-1005-428e-8c1d-0d181c3f2623]  
swh-vault-worker_1            | [2019-04-26 13:11:42,455: INFO/ForkPoolWorker-1] Loading config file /cooker.yml
swh-vault-worker_1            | [2019-04-26 13:19:21,483: ERROR/ForkPoolWorker-1] Bundle cooking failed.
swh-vault-worker_1            | Traceback (most recent call last):
swh-vault-worker_1            |   File "/srv/softwareheritage/venv/lib/python3.6/site-packages/swh/vault/cookers/base.py", line 122, in cook
swh-vault-worker_1            |     self.backend.put_bundle(self.CACHE_TYPE_KEY, self.obj_id, bundle)
swh-vault-worker_1            |   File "/srv/softwareheritage/venv/lib/python3.6/site-packages/swh/vault/api/client.py", line 45, in put_bundle
swh-vault-worker_1            |     data=bundle)
swh-vault-worker_1            |   File "/srv/softwareheritage/venv/lib/python3.6/site-packages/swh/core/api/__init__.py", line 198, in post
swh-vault-worker_1            |     return self._decode_response(response)
swh-vault-worker_1            |   File "/srv/softwareheritage/venv/lib/python3.6/site-packages/swh/core/api/__init__.py", line 235, in _decode_response
swh-vault-worker_1            |     response.content,
swh-vault-worker_1            | swh.core.api.RemoteException: Unexpected status code for API request: 413 (b'Maximum request body size 1048576 exceeded, actual body size 1272960')

Diff Detail

Repository
rDVAU Software Heritage Vault
Branch
fix-client-max-size
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 5580
Build 7601: tox-on-jenkinsJenkins
Build 7600: arc lint + arc unit

Event Timeline

Shouldn't the client_max_size be added to DEFAULT_CONFIG instead?

Hmm, it is already. Shouldn't cfg['client_max_size'] always work, then?

I know it is already set in default config but I preferred to explicitly set a good default value in case where a config file is used and is missing the client_max_size entry (as Idid in swh-objstorage here : rDOBJSc334336b3ea7a068efa1dfe9c667349eef02f57c).
We met so many errors by the past when this important parameter is not provided to aiohttp, so we should have a fallback value to avoid them.

Forget my last comment, reading the value using cfg['client_max_size'] is indeed enough here. I will update accordingly.

Update: client_max_size is already set with a good default value in default configuration, no need to duplicate it

douardda added a subscriber: douardda.

Not a big stuff, but I would have preferred to see the refactoring (of the get_local_backend's code) and the actual fix (add client_max_size arg when calling make_app) in 2 revisions.

This revision is now accepted and ready to land.May 7 2019, 2:32 PM

Update: separate introduced changes into two different commits

This revision was automatically updated to reflect the committed changes.