There remained errors on the pypi loader about None reference.
Sample stacktrace for origin https://pypi.org/project/configpy/
[2018-11-29 08:38:27,300: ERROR/Worker-1903] Loading failure, updating to `partial` status
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 886, in load
self.prepare(*args, **kwargs)
File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 152, in prepare
self._prepare_state()
File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 161, in _prepare_state
self.known_artifacts = self._known_artifacts(last_snapshot)
File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 115, in _known_artifacts
for rev in last_snapshot['branches'].values()
File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 116, in <listcomp>
if rev['target_type'] == 'revision']
TypeError: 'NoneType' object is not subscriptablekibana dashboard is at [1]
Its snapshot [2] feels wrong:
{'branches': {b'HEAD': {'target': b'releases/0.5', 'target_type': 'alias'},
b'releases/0.2': None,
b'releases/0.3': None,
b'releases/0.4': None,
b'releases/0.5': {'target': b'\xa5\x978\x92;S{\xf6\xcfE\xa9r'
b'\xf0\xdd\x85e\xe8Z\x1d\x80',
'target_type': 'revision'}},
'id': b'\xee<\xdfq\xc76\xec\xa2Dn?\x8aE\xb3\xc3\xe0I\x87H\xb0',
'next_branch': None}[2] Retrieving snapshot:
c = {'storage': {
'cls': 'remote',
'args': {'url': 'http://uffizi.internal.softwareheritage.org:5002/'}
}
}
from swh.storage import get_storage
storage = get_storage(**c['storage']
origin = storage.origin_get({'type': 'pypi', 'url': 'https://pypi.org/project/configpy/'}
origin_id = origin['id']
snap = s.snapshot_get_latest(origin_id)
from pprint import pprint
pprint(snap)Note:
D710 fixed a wrong behavior in snapshot resolution (most probably few snapshots were really used prior to this fix).
-> This led to new errors fixed in D727 (which catch most errors when trying to solve the snapshot revisions)
-> This finally revealed that some snapshots are badly formatted (i did not expect snapshot branches to target None)