Page MenuHomeSoftware Heritage

Deposit: error when submitting through cli
Closed, MigratedEdits Locked

Description

Currently:

$ git clone https://github.com/SoftwareHeritage/swh-deposit.git swh
$ cd swh && ./setup build && ./setup install

and trying to upload like this:

swh-deposit deposit --username intel --password <redacted> --archive linuxperf-v4.3.1.27.tar.gz --metadata
linuxperf-v4.3.1.27.metadata.xml --partial
[08/Apr/2019 11:35:03] DEBUG [swh.deposit.cli:418] Parsing cli options
xml:b'<?xml version="1.0" encoding="utf-8"?>\n<sword:error xmlns="http://www.w3.org/2005/Atom"\n      
xmlns:sword="http://purl.org/net/sword/">\n    <summary>Invalid username/password.</summary>\n   
<sword:treatment>processing failed</sword:treatment>\n    \n    <sword:verboseDescription>\n        API is protected
by basic authentication\n    </sword:verboseDescription>\n    \n</sword:error>\n'
[08/Apr/2019 11:35:04] DEBUG [swh.deposit.cli:326] Create deposit
xml:b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>502 Bad
Gateway</title>\n</head><body>\n<h1>Bad Gateway</h1>\n<p>The proxy server received an invalid\r\nresponse from an
upstream server.<br />\r\n</p>\n</body></html>\n'
Traceback (most recent call last):
  File "/usr/local/bin/swh-deposit", line 11, in <module>
    load_entry_point('swh.deposit==0.0.67.post14', 'console_scripts', 'swh-deposit')()
  File "/usr/local/lib/python3.7/site-packages/swh.deposit-0.0.67.post14-py3.7.egg/swh/deposit/cli.py", line 449, in
main
    return cli(auto_envvar_prefix='SWH_DEPOSIT')
  File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/Click-7.0-py3.7.egg/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/swh.deposit-0.0.67.post14-py3.7.egg/swh/deposit/cli.py", line 443, in
deposit
    r = deposit_create(config, dry_run, logger)
  File "/usr/local/lib/python3.7/site-packages/swh.deposit-0.0.67.post14-py3.7.egg/swh/deposit/cli.py", line 336, in
deposit_create
    metadata_path, in_progress, logger)
  File "/usr/local/lib/python3.7/site-packages/swh.deposit-0.0.67.post14-py3.7.egg/swh/deposit/client/__init__.py",
line 540, in deposit_create
    slug)
  File "/usr/local/lib/python3.7/site-packages/swh.deposit-0.0.67.post14-py3.7.egg/swh/deposit/client/__init__.py",
line 291, in execute
    error = self.parse_result_error(r.text)
  File "/usr/local/lib/python3.7/site-packages/swh.deposit-0.0.67.post14-py3.7.egg/swh/deposit/client/__init__.py",
line 255, in parse_result_error
    return _parse_with_filter(xml_content, keys=['summary', 'detail'])
  File "/usr/local/lib/python3.7/site-packages/swh.deposit-0.0.67.post14-py3.7.egg/swh/deposit/client/__init__.py",
line 54, in _parse_with_filter
    data = _parse(stream, encoding=encoding)
  File "/usr/local/lib/python3.7/site-packages/swh.deposit-0.0.67.post14-py3.7.egg/swh/deposit/client/__init__.py",
line 34, in _parse
   data = xmltodict.parse(stream, encoding=encoding, process_namespaces=False)
  File "/usr/local/lib/python3.7/site-packages/xmltodict-0.12.0-py3.7.egg/xmltodict.py", line 328, in parse
    parser.Parse(xml_input, True)
xml.parsers.expat.ExpatError: syntax error: line 1, column 49

It looks like I am missing correct authentication:

b'<?xml version="1.0" encoding="utf-8"?>\n<sword:error xmlns="http://www.w3.org/2005/Atom"\n      
xmlns:sword="http://purl.org/net/sword/">\n    <summary>Invalid username/password.</summary>\n   
<sword:treatment>processing failed</sword:treatment>\n    \n    <sword:verboseDescription>\n        API is protected
by basic authentication\n    </sword:verboseDescription>\n    \n</sword:error>\n'
b'<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">\n<html><head>\n<title>502 Bad
Gateway</title>\n</head><body>\n<h1>Bad Gateway</h1>\n<p>The proxy server received an invalid\r\nresponse from an
upstream server.<br />\r\n</p>\n</body></html>\n'

So could you please advise how to proceed?

Event Timeline

ardumont renamed this task from error when submitting a deposit through the cli to deposit: error when submitting through cli.Apr 10 2019, 1:07 PM
ardumont triaged this task as High priority.
ardumont created this task.
ardumont renamed this task from deposit: error when submitting through cli to Deposit: error when submitting through cli.Apr 14 2019, 5:15 PM

I have badly dealt with this issue.

I'm noting this for next time.

A posteriori, in the end, i think there were multiple issues both side.

I should have asked multiple things to help analyze the issue:

  • the metadata file linuxperf-v4.3.1.27.metadata.xml (as per the parsing error xml.parsers.expat.ExpatError: syntax error: line 1, column 49 implies something badly formatted).
  • if there is a proxy in the chain (as per the Bad Gateway implies)

In any case, there was also an issue with the --collection flag which is optional.
Which means that we expect the cli to fetch that information alone (to get out of the way of the client) but that failed at the time. This is currently fixed with D1411.

There is no client misconfiguration (regarding authentication) though as this was not reproduced (with same credential).

As soon as i land the following improvments (which derived from the new use cases) D1416, D1419, D1421, deploy, i'll notify the client and close this.

Cheers,

ardumont lowered the priority of this task from High to Normal.Apr 17 2019, 10:02 AM

the metadata file linuxperf-v4.3.1.27.metadata.xml (as per the parsing error xml.parsers.expat.ExpatError: syntax error: line 1, column 49 implies something badly formatted).

Indeed, i was able to reproduce the error by providing a wrongly formatted metadata xml file.
Which explodes in the server side.

Will fix that by returning a 400 (bad request) in that case.