Right now, the new archive.softwareheritage.org running django has been deployed with the rate limit backend as an in-memory one (default is LocMemCache).
This will pose a problem regarding the rate limit being shared between all gunicorn workers (which should not).
Prior to this deployment, we used redis (when using Flask_Limiter).
We should be running an equivalent system to avoid such shortcomings.
According to djangorestframework/django, we could use memcached.
Discussion:
14:18:34 ardumont | what do i do with the redis instance that flask limiter used? 14:49:02 zack | where are the live counters stored in the new implementation? 14:49:08 zack | anyway, if it's no longer needed, it should go away 14:50:29 ardumont | i don't think it's stored anymore (NDA: I meant not persistent :) 14:50:32 ardumont | and not in redis anyway 14:50:39 ardumont | anlambert: do you confirm? 14:50:48 ardumont | other than that i'm ready to deploy 14:50:54 ardumont | i fixed the tiny errors in the manifest 14:52:18 olasd | it uses django's caching mechanism 14:55:29 olasd | so, by default, a local memory cache 14:58:31 olasd | which means that the rate limit would be spread out across workers 14:58:57 olasd | which is fine for a first deployment but needs to be fixed eventually 15:00:07 ardumont | you mean the different gunicorn workers? 15:00:10 olasd | yes 15:00:47 olasd | each gunicorn worker would get its own separate counter 15:01:20 ardumont | ack, thx (i was reading the doc, for info http://www.django-rest-framework.org/api-guide/throttling/#setting-up-the-cache) 15:01:31 ardumont | (not for you olasd :) 15:02:58 zack | so, maybe, before throwing away redis, check if it is still a viable backend? 15:04:01 olasd | stretch and up have python3-django-redis 15:04:04 olasd | not jessie 15:04:22 ardumont | ah, nice to know 15:04:51 ardumont | i won't throw away redis yet :) 15:04:52 olasd | but I'm not married to redis 15:06:04 ardumont | zack: it's not clear if it is a blocker to deploy (as olasd said, we can fix it later, as in next week for example) 15:06:18 zack | i didn't say it was a blocker :) 15:06:26 ardumont | ok, thx for clarifying :) 15:07:14 olasd | rate limiting is the only reason for redis to be deployed on our infra, we can just as well replace it with memcached if that's what django prefers 15:07:24 olasd | (flask-limiter also supported memcached) 15:07:37 ardumont | yes, and memcached is in the default list i refer to 15:08:07 ardumont | (well, i'd need to check for the version though, we're still in 1.7 in oldstable :) 15:08:19 ardumont | and the link is 1.11 15:08:45 ardumont | (it is present in 1.7 :)