Page MenuHomeSoftware Heritage

Requests to the scheduler API take up to 6 seconds in prod
Open, HighPublic

Description

Right now, this command take 2.5s to execute:

echo 'indexer_origin_head;oneshot;[[1]];{"policy_update": "update-dups", "parse_ids": false}' | time python3 -m swh.scheduler.cli --cls remote --url http://saatchi.internal.softwareheritage.org:5008/ task schedule /dev/stdin -c type -c policy -c args -c kwargs -d ';'

Yesterday, it was taking 5s. I believe this is because I scheduled a batch of 100k tasks (that is now done); with this command:

python3 -c 'for x in range(100000, 200000): print("""indexer_origin_head;oneshot;[[%d]];{"policy_update": "update-dups", "parse_ids": false}""" % x)'  | python3 -m swh.scheduler.cli --cls remote --url http://saatchi.internal.softwareheritage.org:5008/ task schedule /dev/stdin -c type -c policy -c args -c kwargs -d ';'

Event Timeline

vlorentz created this task.Jan 23 2019, 1:18 PM
vlorentz triaged this task as High priority.
ardumont renamed this task from Requests to the scheduler AI take up to 6 seconds in prod to Requests to the scheduler API take up to 6 seconds in prod.Jan 24 2019, 10:39 AM