Page MenuHomeSoftware Heritage
Create Task
Maniphest T2300

swh-scanner: print a nicer error message when rate limit is hit
Closed, MigratedEdits Locked
Actions

Description

currently, when that happens, a nasty traceback is spit out, ending with something like this:

swh.scanner.exceptions.APIError: API Error: "<bound method ClientResponse.text of <ClientResponse(https://archive.softwareheritage.org/api/1/known/) [429 Too Many Requests]>
<CIMultiDictProxy('Date': 'Fri, 28 Feb 2020 16:05:34 GMT', 'Server': 'gunicorn/19.9.0', 'Content-Type': 'application/json', 'Retry-After': '3507', 'Vary': 'Accept,Cookie', 'Allow': 'POST, OPTIONS', 'X-RateLimit-Limit': '120', 'X-RateLimit-Remaining': '0', 'X-RateLimit-Reset': '1582909441', 'X-Frame-Options': 'SAMEORIGIN', 'Content-Length': '71', 'Via': '1.1 archive.softwareheritage.org', 'Access-Control-Allow-Origin': '*', 'X-Varnish': '52209502', 'Age': '0', 'Via': '1.1 varnish (Varnish/6.1)', 'Strict-Transport-Security': 'max-age=15768000;', 'Connection': 'keep-alive')>
> with given values ['swh:1:dir:a0de650e15e7e1a65c459784b20851d24f945c9a', 'swh:1:dir:d1b8a4259238f327399a9533a70bbf4a4f5b612d', 'swh:1:cnt:f14e196529632bc4e37f7e7deb0b49db4742a423']"

we should output a nice(r) error message instead

Revisions and Commits

rDTSCN Code scanner
D2794rDTSCNcb5faf6e866a exceptions: APIError response

Related Objects

Event Timeline

zack assigned this task to DanSeraf.Feb 28 2020, 5:06 PM
zack triaged this task as Low priority.
zack created this task.
zack added a project: Easy hack.
DanSeraf mentioned this in D2794: exceptions: custom error response.Mar 10 2020, 12:11 PM
DanSeraf closed this task as Resolved by committing rDTSCNcb5faf6e866a: exceptions: APIError response.Mar 11 2020, 9:51 AM
DanSeraf added a commit: rDTSCNcb5faf6e866a: exceptions: APIError response.
zack reopened this task as Open.Apr 15 2020, 1:55 PM

Reopening, as it looks like this issue is back, e.g.:

$ swh scanner scan -f sunburst . 
Traceback (most recent call last):
  File "/home/zack/.virtualenvs/swh/bin/swh", line 11, in <module>
    load_entry_point('swh.core', 'console_scripts', 'swh')()
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-core/swh/core/cli/__init__.py", line 122, in main
    return swh(auto_envvar_prefix="SWH")
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/home/zack/.virtualenvs/swh/lib/python3.7/site-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/cli.py", line 56, in scan
    loop.run_until_complete(run(path, api_url, source_tree))
  File "/usr/lib/python3.7/asyncio/base_events.py", line 587, in run_until_complete
    return future.result()
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 138, in run
    await _scan(root, session, api_url, source_tree)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 135, in _scan
    await _scan(path, session, api_url, source_tree)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 135, in _scan
    await _scan(path, session, api_url, source_tree)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 135, in _scan
    await _scan(path, session, api_url, source_tree)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 125, in _scan
    for path, pid, found in await parse_path(root, session, api_url):
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 104, in parse_path
    parsed_pids = await pids_discovery(list(parsed_paths.values()), session, api_url)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 61, in pids_discovery
    return await make_request(pids)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/scanner.py", line 49, in make_request
    error_response(resp.reason, resp.status, endpoint)
  File "/home/zack/dati/projects/sw-heritage/git/swh-environment/swh-scanner/swh/scanner/exceptions.py", line 18, in error_response
    raise APIError(error_msg)
swh.scanner.exceptions.APIError: "429 Too Many Requests: 'https://archive.softwareheritage.org/api/1/known/'"
zack renamed this task from scanner: print a nicer error message when rate limit is hit to swh-scanner: print a nicer error message when rate limit is hit.Sep 8 2020, 10:24 AM
zack removed DanSeraf as the assignee of this task.Dec 19 2020, 9:48 PM
zack added a subscriber: DanSeraf.
pranay101 added a subscriber: pranay101.Apr 23 2022, 8:59 AM

Hey there. Is this issue still open? I'd like to work on it

gitlab-migration closed this task as Migrated.Jan 8 2023, 5:01 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

About · Developer information · Legal