Page MenuHomeSoftware Heritage

Open up the Software Heritage API to the world
Closed, MigratedEdits Locked

Description

It's about time we open up our (HTTP-, JSON-based) API to the world.

We need to make very sure that we don't get our pants caught on fire as we do it: we must avoid having our infrastructure torn down by the load.

This meta task tracks the steps we need to take to be able to do this.

  • (critical) create a read-only replica database (this is a by-product of T615)
  • (critical) implement rate limiting for anonymous access to the API (T616)
  • (normal) token-based user authentication and rate-limiting for access to the API
  • (normal) split moma in two virtual machines: one for the scheduler and associated stuff (rabbitmq) and one for the web ui

While opening up the API for unauthenticated requests should not be an issue if we have a second database for read-only access, we should ramp up the load by allowing only heavily rate-limited API access at first, rather than open it all up and see it explode mid-flight.

Provisions for authenticated access will allow us to give special rate limits to the people that need it, and be able to finely control it. This is not critical to have in the beginning, but we should be ready to provide it ASAP anyway. Might as well do that while we're adding the rate-limiting stuff.

Splitting the scheduling and the Web UI should have been done long ago but is now easy as pie as we have another hypervisor, it just needs the work to be done.

Event Timeline

zack added a project: Restricted Project.Jan 11 2017, 4:38 PM
zack moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
olasd changed the status of subtask T618: debian packaging of flask-limiter from Open to Work in Progress.Jan 11 2017, 5:53 PM
zack created subtask Unknown Object (Maniphest Task).Jan 12 2017, 10:24 AM
ardumont closed subtask Unknown Object (Maniphest Task) as Resolved.Jan 24 2017, 11:20 AM
zack changed the status of subtask T621: content review for the API doc page from Open to Work in Progress.Jan 30 2017, 4:54 PM
zack claimed this task.
zack added a subscriber: zack.

*drum roll*
*tsssss*

gitlab-migration changed the status of subtask T618: debian packaging of flask-limiter from Resolved to Migrated.
gitlab-migration changed the status of subtask T653: Split the scheduler (rabbitmq) out of the moma virtual machine from Resolved to Migrated.
gitlab-migration changed the status of subtask T655: tweak HTTP auth for first API public release from Resolved to Migrated.
gitlab-migration changed the status of subtask T621: content review for the API doc page from Resolved to Migrated.
gitlab-migration changed the status of subtask Unknown Object (Maniphest Task) from Resolved to Migrated.