DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /origin/visit/get_latest HTTP/1.1" 400 74
ERROR:swh.loader.pypi.PyPILoader:Loading failure, updating to `partial` status
Traceback (most recent call last):
  File "/home/archit/swh-environment/swh-loader-core/swh/loader/core/loader.py", line 868, in load
    self.prepare(*args, **kwargs)
  File "/home/archit/swh-environment/swh-loader-pypi/swh/loader/pypi/loader.py", line 151, in prepare
    self._prepare_state()
  File "/home/archit/swh-environment/swh-loader-pypi/swh/loader/pypi/loader.py", line 159, in _prepare_state
    last_snapshot = self._last_snapshot()
  File "/home/archit/swh-environment/swh-loader-pypi/swh/loader/pypi/loader.py", line 131, in _last_snapshot
    self.origin, require_snapshot=True)
  File "/home/archit/swh-environment/swh-storage/swh/storage/api/client.py", line 207, in origin_visit_get_latest
    'require_snapshot': require_snapshot})
  File "/home/archit/swh-environment/swh-core/swh/core/api/__init__.py", line 205, in post
    return self._decode_response(response)
  File "/home/archit/swh-environment/swh-core/swh/core/api/__init__.py", line 237, in _decode_response
    raise pickle.loads(decode_response(response))
psycopg2.ProgrammingError: can't adapt type 'dict'
DEBUG:swh.loader.pypi.PyPILoader:Updating origin_visit for origin https://pypi.org/pypi/numpy/ with status partial
DEBUG:urllib3.connectionpool:http://localhost:5002 "POST /origin/visit/update HTTP/1.1" 200 1




The script I used


import logging
logging.basicConfig(level=logging.DEBUG)

from swh.loader.pypi.loader import PyPILoader

project='numpy'

PyPILoader().load(project, 'https://pypi.org/pypi/%s/' % project, project_metadata_url= 'https://pypi.org/pypi/%s/json' % project)



swh-storage logs


swh-storage_1                 | [2019-07-11 13:27:19 +0000] [39] [DEBUG] POST /origin/visit/get_latest
swh-storage_1                 | ERROR:root:can't adapt type 'dict'
swh-storage_1                 | Traceback (most recent call last):
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
swh-storage_1                 |     rv = self.dispatch_request()
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
swh-storage_1                 |     return self.view_functions[rule.endpoint](**req.view_args)
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 36, in d
swh-storage_1                 |     return f(*a, **kw)
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/api/server.py", line 417, in origin_visit_get_latest
swh-storage_1                 |     get_storage().origin_visit_get_latest(**decode_request(request)))
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/db/common.py", line 49, in _meth
swh-storage_1                 |     return meth(self, *args, db=db, cur=cur, **kwargs)
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/storage.py", line 1408, in origin_visit_get_latest
swh-storage_1                 |     origin = self.origin_get({'url': origin}, db=db, cur=cur)
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/db/common.py", line 41, in _meth
swh-storage_1                 |     ret = meth(self, *args, **kwargs)
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/storage.py", line 1509, in origin_get
swh-storage_1                 |     for result in results]
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/storage.py", line 1508, in <listcomp>
swh-storage_1                 |     results = [dict(zip(self.origin_keys, result))
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/storage/db.py", line 675, in origin_get_with
swh-storage_1                 |     cur, query, origins)
swh-storage_1                 |   File "/srv/softwareheritage/venv/lib/python3.7/site-packages/swh/core/db/db_utils.py", line 145, in execute_values_generator
swh-storage_1                 |     parts.append(cur.mogrify(template, args))
swh-storage_1                 | psycopg2.ProgrammingError: can't adapt type 'dict'
swh-storage_1                 | [2019-07-11 13:27:19 +0000] [39] [DEBUG] POST /origin/visit/update