Page MenuHomeSoftware Heritage

api/sync: allow to configure requests for pool size and retries
ClosedPublic

Authored by douardda on Jun 28 2019, 12:16 PM.

Details

Summary

by default:

  • increase the connection pool size and
  • set retry to 3 (number of retries each connection should attempt).

These config params can be set directly from config files for 'remote'
api clients. For example, to specify these in an objstorage client
config file:

objstorage:
  cls: remote
  args:
    url: swh-objstorage:5003
    max_retries: 5
    pool_connections: 100

This is needed for example for swh-journal's content-replayer to be able to spawn more then 10 worker threads.

Diff Detail

Repository
rDCORE Foundations and core functionalities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont added a subscriber: ardumont.

I'm not sure we want the https adapter part but other than that, this sound good.

I took at look at the doc [1].

[1] https://2.python-requests.org/en/master/user/advanced/#id15

swh/core/api/__init__.py
166

I don't think we use https anywhere for those yet so yagni? ;)

This revision is now accepted and ready to land.Jun 28 2019, 9:56 PM

do only mount the adapter on expected url

as ardumont suggests implicitely ;-)

This revision was automatically updated to reflect the committed changes.