Page MenuHomeSoftware Heritage

Only filters snapshot's revisions to retrieve artifact
ClosedPublic

Authored by ardumont on Wed, Nov 28, 6:34 PM.

Details

Summary

In current production, there are multiple issues about this:

Nov 28 13:02:40 worker15 python3[8850]: [2018-11-28 13:02:40,897: INFO/MainProcess] Received task: swh.loader.pypi.tasks.LoadPyPI[27a370b7-92b3-49ea-b307-4f1c5c9c8597]
Nov 28 13:02:41 worker15 python3[8883]: [2018-11-28 13:02:41,254: ERROR/Worker-10] Loading failure, updating to `partial` status
                                        Traceback (most recent call last):
                                          File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 886, in load
                                            self.prepare(*args, **kwargs)
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 147, in prepare
                                            self._prepare_state()
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 156, in _prepare_state
                                            self.known_artifacts = self._known_artifacts(last_snapshot)
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 117, in _known_artifacts
                                            if 'original_artifact' in revision['metadata']:
                                        TypeError: 'NoneType' object is not subscriptable

That's why the loader got a speed boost earlier this afternoon (T.T)

;)

Test Plan

tox

Diff Detail

Repository
rDLDPY PyPI loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Wed, Nov 28, 6:34 PM

That's why the loader got a speed boost earlier this afternoon (T.T)

Still, after this fix is deployed, the speed boost should hold.

olasd accepted this revision.Wed, Nov 28, 6:40 PM
This revision is now accepted and ready to land.Wed, Nov 28, 6:40 PM
This revision was automatically updated to reflect the committed changes.