Page MenuHomeSoftware Heritage

package.loader: Skip artifacts with no intrinsic metadata
ClosedPublic

Authored by ardumont on Jan 24 2020, 10:46 AM.

Details

Summary

no intrinsic metadata as in no corresponding metadata file (eg. pypi: PKG-INFO, npm: package.json).
Both pypi and npm loaders demonstrate that behavior.

Currently, this means the visit fails during ingestion.
By skipping those artifacts, we can have a more complete ingestion
of the impacted origins (providing other artifacts are correctly configured, with their intrinsic metadata file).

[1] https://sentry.softwareheritage.org/share/issue/4a2d3c5f6d4942ddb94a68bb532b6d02/

[2] https://sentry.softwareheritage.org/share/issue/8773eb3140b94675a08143935a61087c/

Test Plan

tox

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Jan 24 2020, 10:46 AM
ardumont edited the summary of this revision. (Show Details)Jan 24 2020, 10:48 AM
ardumont edited the summary of this revision. (Show Details)Jan 24 2020, 2:42 PM
vlorentz requested changes to this revision.Jan 27 2020, 1:58 PM
vlorentz added a subscriber: vlorentz.

Could you add tests?

This revision now requires changes to proceed.Jan 27 2020, 1:58 PM
ardumont updated this revision to Diff 9253.Jan 28 2020, 12:06 PM

Add tests on both loaders

vlorentz accepted this revision.Jan 28 2020, 12:22 PM

You should replace "intrinsic_metadata" with "extrinsic_metadata". But other than that, ok for me

This revision is now accepted and ready to land.Jan 28 2020, 12:22 PM

My bad, you don't need that