Page MenuHomeSoftware Heritage
Differential D727

Only filters snapshot's revisions to retrieve artifact
ClosedPublic
Actions

Authored by ardumont on Nov 28 2018, 6:34 PM.

Details

Reviewers
anlambert
olasd
Group Reviewers
Reviewers
Commits
rDLDPY00a68fe9ec91: pypi.loader: Only retrieve the snapshot's revisions
rDLDPY4b3edf3518a3: pypi.loader: Fix edge case about revision_get
Summary

In current production, there are multiple issues about this:

Nov 28 13:02:40 worker15 python3[8850]: [2018-11-28 13:02:40,897: INFO/MainProcess] Received task: swh.loader.pypi.tasks.LoadPyPI[27a370b7-92b3-49ea-b307-4f1c5c9c8597]
Nov 28 13:02:41 worker15 python3[8883]: [2018-11-28 13:02:41,254: ERROR/Worker-10] Loading failure, updating to `partial` status
                                        Traceback (most recent call last):
                                          File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 886, in load
                                            self.prepare(*args, **kwargs)
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 147, in prepare
                                            self._prepare_state()
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 156, in _prepare_state
                                            self.known_artifacts = self._known_artifacts(last_snapshot)
                                          File "/usr/lib/python3/dist-packages/swh/loader/pypi/loader.py", line 117, in _known_artifacts
                                            if 'original_artifact' in revision['metadata']:
                                        TypeError: 'NoneType' object is not subscriptable

That's why the loader got a speed boost earlier this afternoon (T.T)

;)

Test Plan

tox

Diff Detail

Repository
rDLDPY PyPI loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Nov 28 2018, 6:34 PM
swh-public-ci added a subscriber: swh-public-ci.Nov 28 2018, 6:36 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDPY/job/tox/9/ for more details.

Harbormaster completed remote builds in B2665: Diff 2288.Nov 28 2018, 6:36 PM
ardumont added a comment.Nov 28 2018, 6:37 PM

That's why the loader got a speed boost earlier this afternoon (T.T)

Still, after this fix is deployed, the speed boost should hold.

olasd accepted this revision.Nov 28 2018, 6:40 PM
This revision is now accepted and ready to land.Nov 28 2018, 6:40 PM
Closed by commit rDLDPY4b3edf3518a3: pypi.loader: Fix edge case about revision_get (authored by ardumont). · Explain WhyNov 28 2018, 6:41 PM
This revision was automatically updated to reflect the committed changes.
ardumont mentioned this in T1396: loader-pypi: Snapshot with null branch are badly handled by loader.Nov 29 2018, 10:05 AM

Revision Contents
Changeset List

PathSize
swh/
loader/
pypi/
2 lines

Diff 2289

View Options

swh/loader/pypi/loader.py

Loading...
About · Developer information · Legal