Page MenuHomeSoftware Heritage

PyPI loader fails when the metadata is incomplete
Closed, ResolvedPublic

Description

Some packages, for instance pycbrf, don't have a full set of metadata.

This makes their loading fail:

Sep 21 14:15:57 worker01 python3[22223]: [2018-09-21 14:15:57,566: INFO/MainProcess] Received task: swh.loader.pypi.tasks.LoadPyPI[ed442f86-5740-4f66-a411-338413cf30be]
Sep 21 14:17:54 worker01 python3[1627]: [2018-09-21 14:17:54,352: ERROR/Worker-108] Loading failure, updating to `partial` status
                                        Traceback (most recent call last):
                                          File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 889, in load
                                            more_data_to_fetch = self.fetch_data()
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 183, in fetch_data
                                            data = next(self.new_artifacts)
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/client.py", line 462, in download_new_releases
                                            yield pkginfo, author(pkginfo), release, artifact, dir_path
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/converters.py", line 51, in author
                                            name = data['author']
                                        KeyError: 'author'
Sep 21 14:17:54 worker01 python3[22223]: [2018-09-21 14:17:54,682: INFO/MainProcess] Task swh.loader.pypi.tasks.LoadPyPI[ed442f86-5740-4f66-a411-338413cf30be] succeeded in 116.92293047503335s: {'status': 'failed'}