Page MenuHomeSoftware Heritage

gitlab-lister: Investigate rate limit implementation further
Open, NormalPublic

Description

It's been implemented and so far potentially used for the gitlab.com main instance.
It's based on their response's headers [1]:

RateLimit-Limit: 600
RateLimit-Observed: 1
RateLimit-Remaining: 599
RateLimit-Reset: 1561888426
RateLimit-ResetTime: Sun, 30 Jun 2019 09:53:46 GMT

Check other instances (salsa's debian for example) for a similar use pattern. [1]

[1] Checking a head query on salsa does not render those:

curl -I https://salsa.debian.org/api/v4/projects/                                                                                                                                                                                                                                                                                                                    ~
HTTP/1.1 200 OK
Date: Sun, 30 Jun 2019 09:52:11 GMT
Server: Apache
Referrer-Policy: no-referrer
X-Xss-Protection: 1
Strict-Transport-Security: max-age=15552000
Public-Key-Pins: pin-sha256="JX3rnmIs7tgG+zN5HNBXaA8rfi6RKVlvOPGnn+uJ/xI="; pin-sha256="5xBLLloCLM71PnTWnZuxgQmaW3ziZhvPuSBbwKI3D3o="; max-age=5184000
Cache-Control: no-cache
Content-Length: 18197
Content-Type: application/json
Link: <https://salsa.debian.org/api/v4/projects?membership=false&order_by=created_at&owned=false&page=2&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="next", <https://salsa.debian.org/api/v4/projects?membership=false&order_by=created_at&owned=false&page=1&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="first", <https://salsa.debian.org/api/v4/projects?membership=false&order_by=created_at&owned=false&page=1591&per_page=20&simple=false&sort=desc&starred=false&statistics=false&with_custom_attributes=false&with_issues_enabled=false&with_merge_requests_enabled=false>; rel="last"
Vary: Origin
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-Next-Page: 2
X-Page: 1
X-Per-Page: 20
X-Prev-Page:
X-Request-Id: xKLWNvr19L5
X-Runtime: 0.505211
X-Total: 31818
X-Total-Pages: 1591
X-Clacks-Overhead: GNU Terry Pratchett

[1] https://forge.softwareheritage.org/source/swh-lister/browse/master/swh/lister/gitlab/lister.py$44-55

Event Timeline

ardumont triaged this task as Normal priority.

I did not find anything specific to rate limit policy for salsa.
I found an api usage best practices chapter though [1].

[1] https://wiki.debian.org/Salsa/Doc?action=fullsearch&context=180&value=rate+limit&titlesearch=Titles#API_Usage_Best_practises

ardumont updated the task description. (Show Details)Jul 1 2019, 10:04 AM