Page MenuHomeSoftware Heritage

package.loader: Skip artifacts with no intrinsic metadata
ClosedPublic

Authored by ardumont on Jan 24 2020, 10:46 AM.

Details

Summary

no intrinsic metadata as in no corresponding metadata file (eg. pypi: PKG-INFO, npm: package.json).
Both pypi and npm loaders demonstrate that behavior.

Currently, this means the visit fails during ingestion.
By skipping those artifacts, we can have a more complete ingestion
of the impacted origins (providing other artifacts are correctly configured, with their intrinsic metadata file).

[1] https://sentry.softwareheritage.org/share/issue/4a2d3c5f6d4942ddb94a68bb532b6d02/

[2] https://sentry.softwareheritage.org/share/issue/8773eb3140b94675a08143935a61087c/

Test Plan

tox

Diff Detail

Repository
rDLDBASE Generic VCS/Package Loader
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 10267
Build 15234: tox-on-jenkinsJenkins
Build 15233: arc lint + arc unit

Event Timeline

vlorentz added a subscriber: vlorentz.

Could you add tests?

This revision now requires changes to proceed.Jan 27 2020, 1:58 PM

You should replace "intrinsic_metadata" with "extrinsic_metadata". But other than that, ok for me

This revision is now accepted and ready to land.Jan 28 2020, 12:22 PM