Page MenuHomeSoftware Heritage

web API rate limit: 10x more quota for authenticated users
Closed, ResolvedPublic

Description

We are seeing an increase in rate limit lifting requests, which is great but also a pain to handle. As currently the default rate limit for non-authenticated users is really draconian (and arbitrary), we would like to have a 10x higher quota for all authenticated users. This will relieve the pressure on handling requests and also encourage both user registration and authenticated use of the API, which are both good things. According to past discussions (cc: @olasd) that should also not significantly impact the infra load (and even if it does, we want to do some real-world capacity testing and see when problems strikes, so this will be a good experiment too).

@anlambert: can we have 10x more rate limit quota by default for all authenticated users?

(of course it should still be possible to have a non-default quota for specific authenticated users, usually but not necessarily higher than the default one)

Event Timeline

zack triaged this task as High priority.May 7 2021, 9:35 AM
zack created this task.

@anlambert ; ping me when this is done, so we can answer some pending requests :-)

anlambert changed the task status from Open to Work in Progress.May 7 2021, 10:58 AM

@zack, @rdicosmo yes this is totally feasible by adding a dedicated Django REST Framework throttling handler for authenticated users.

Let's work on that then.

@zack, @rdicosmo yes this is totally feasible by adding a dedicated Django REST Framework throttling handler for authenticated users.

Let's work on that then.

Thanks! Please make sure we can also have specific rate limits for specific classees of authenticated users, on top of the default new rate limit for all registered users.

Thanks! Please make sure we can also have specific rate limits for specific classees of authenticated users, on top of the default new rate limit for all registered users.

We already have a specific user permission to lift rate limit for specific users that will remain in place.
If we need to tune rate limit for specific type of users, this could be easily added in the new throttling
code I am currently working on.

If we need to tune rate limit for specific type of users, this could be easily added in the new throttling
code I am currently working on.

Nice! I expect we will have various types/categories of users to handle... for the moment, already "ambassadors" and "publishers"

anlambert claimed this task.

The new rate limit policy has been implemented and deployed to production, see X-RateLimit-Limit header value below.

Anonymous user
14:45 $ curl -i https://archive.softwareheritage.org/api/1/stat/counters/
HTTP/1.1 200 OK
Date: Fri, 07 May 2021 12:58:41 GMT
Server: gunicorn/19.9.0
Content-Type: application/json
Vary: Accept,Cookie,Accept-Encoding
Allow: OPTIONS, OPTIONS, HEAD, GET
X-RateLimit-Limit: 120
X-RateLimit-Remaining: 119
X-RateLimit-Reset: 1620392351
X-Frame-Options: SAMEORIGIN
Via: 1.1 archive.softwareheritage.org
X-Varnish: 11580013
Age: 0
Via: 1.1 varnish (Varnish/6.1)
Strict-Transport-Security: max-age=15768000;
Accept-Ranges: bytes
Content-Length: 124
Connection: keep-alive

{"origin":156417833,"revision":2128899595,"content":10162254604,"directory":8477359190,"release":17293952,"person":42598760}
Authenticated user
14:59 $ curl -i -H "Authorization: Bearer ${TOKEN}" https://archive.softwareheritage.org/api/1/stat/counters/
HTTP/1.1 200 OK
Date: Fri, 07 May 2021 12:59:29 GMT
Server: gunicorn/19.9.0
Content-Type: application/json
Vary: Accept,Cookie,Accept-Encoding
Allow: OPTIONS, OPTIONS, HEAD, GET
X-RateLimit-Limit: 1200
X-RateLimit-Remaining: 1199
X-RateLimit-Reset: 1620392372
X-Frame-Options: SAMEORIGIN
Content-Length: 124
Via: 1.1 archive.softwareheritage.org
X-Varnish: 11927594
Age: 0
Via: 1.1 varnish (Varnish/6.1)
Strict-Transport-Security: max-age=15768000;
Accept-Ranges: bytes
Connection: keep-alive

{"origin":156417833,"revision":2128899595,"content":10162254604,"directory":8477359190,"release":17293952,"person":42598760}