Page MenuHomeSoftware Heritage
Create Task
Maniphest T652

estimate max load we can withstand for anonymous access to the API
Closed, MigratedEdits Locked
Actions

Description

We need this as a data point for T651.

Locust seems the right tool for this job, as we can use it to generate non trivial(ly cacheable) traffic patterns, including look up of existing and non existing objects.

Related Objects
Search...

Event Timeline

zack created this task.Jan 30 2017, 2:07 PM
zack added a subtask: T615: set up pg_logical-based master/slave replication for the postgres db.
zack moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
zack moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
zack added a project: System administration.Jan 30 2017, 2:10 PM
zack moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.
olasd closed subtask T615: set up pg_logical-based master/slave replication for the postgres db as Resolved.Jan 31 2017, 1:03 PM
olasd added a comment.Jan 31 2017, 5:56 PM

I spent some time looking at locustio

  • released versions are Python2 only
  • there's an alpha that supports python3

I ran it on an external server to test it out, with a simple config that just hits a static page.

  • it maxes out one CPU without using the others
  • it randomly hits another URL ??? (I set it up with /api/, sometimes it would try to access //api ???)

I tested master-slave mode on the same machine and it seems to work, however the random URI problem is more present.

Setting aside locustio, I'll dump a random sample of valid object IDs so we can validate some actual usage patterns that would hit the database.

olasd mentioned this in rDSNIPc6348770a03f: Add locust.io testing data.Feb 1 2017, 12:40 PM
zack changed the task status from Open to Work in Progress.Feb 2 2017, 10:27 AM
zack closed this task as Resolved.Feb 2 2017, 10:36 AM
zack added a subscriber: ardumont.

We have just done (with @olasd and @ardumont) a Locust-based test with 15 slaves.
Result: we bottleneck at ~70 requests/sec at the Apache level (the backend postgres is almost idle).

zack mentioned this in T651: define rate limits for anonymous access to the API.Feb 2 2017, 10:38 AM
olasd mentioned this in rSPSITE448a917692a5: data/defaults: update webui rate limits after load testing (T652).Feb 2 2017, 10:43 AM
gitlab-migration changed the task status from Resolved to Migrated.Oct 19 2022, 5:52 PM
gitlab-migration claimed this task.
gitlab-migration changed the status of subtask T615: set up pg_logical-based master/slave replication for the postgres db from Resolved to Migrated.
gitlab-migration added subscribers: olasd, gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal