Page MenuHomeSoftware Heritage
Differential D7992

npm: Add workaround for mangled package descriptions
ClosedPublic
Actions

Authored by vlorentz on Jun 15 2022, 6:30 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4277: Deal with null characters in the output of the metadata indexer
Commits
rDCIDX62db0cb2086b: npm: Add workaround for mangled package descriptions
Summary

Null bytes in JSON produced by indexers cause the indexer-storage to crash (T4277),
and this case seems to be the only current source of such crashes;
so this should fix the issue for now.

A future commit will sanitize all JSON documents before storage.

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 29885
Build 46717: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 46716: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
702 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.test_metadata.TestMetadata::test_npm_adversarial
self = <swh.indexer.tests.test_metadata.TestMetadata object at 0x7eff1204bda0> @settings(suppress_health_check=[HealthCheck.too_slow])
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.codemeta::swh.indexer.codemeta.merge_values
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.normalize_groupId
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.parse_licenses
1 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.parse_repositories
View Full Test Results (1 Failed · 339 Passed · 15 Skipped)

Event Timeline

vlorentz created this revision.Jun 15 2022, 6:30 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 15 2022, 6:30 PM
vlorentz added a task: T4277: Deal with null characters in the output of the metadata indexer.Jun 15 2022, 6:30 PM
swh-public-ci added a comment.Jun 15 2022, 6:35 PM

Build is green

Patch application report for D7992 (id=28796)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 026e9fb86c138df9f62473a0a03cf1cf9fd1617b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/255/ for more details.

Harbormaster completed remote builds in B29878: Diff 28796.Jun 15 2022, 6:35 PM
vlorentz requested review of this revision.Jun 15 2022, 6:35 PM
vlorentz updated this revision to Diff 28802.Jun 16 2022, 9:58 AM

fix outdated comment

swh-public-ci added a comment.Jun 16 2022, 10:04 AM

Build has FAILED

Patch application report for D7992 (id=28802)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 92fc18eae4c239e0c5811060d706986ea54063f7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/256/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/256/console

Harbormaster failed remote builds in B29885: Diff 28802!Jun 16 2022, 10:04 AM
vlorentz updated this revision to Diff 28803.Jun 16 2022, 10:17 AM

fix None support

vlorentz updated this revision to Diff 28804.Jun 16 2022, 10:21 AM

fix test

swh-public-ci added a comment.Jun 16 2022, 10:23 AM

Build has FAILED

Patch application report for D7992 (id=28803)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 4c1a829a6b3331c89a0af1c1234b58e271c25b52
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/257/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/257/console

Harbormaster failed remote builds in B29886: Diff 28803!Jun 16 2022, 10:23 AM
swh-public-ci added a comment.Jun 16 2022, 10:26 AM

Build is green

Patch application report for D7992 (id=28804)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 38bd1f45cda2599730821b07d446c9c37f0d8af1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/258/ for more details.

Harbormaster completed remote builds in B29887: Diff 28804.Jun 16 2022, 10:26 AM
ardumont accepted this revision.Jun 16 2022, 10:31 AM
ardumont added a subscriber: ardumont.

lgtm, one remark inline.

swh/indexer/metadata_dictionary/npm.py
183

you're missing that one in your docstring sample tests.

This revision is now accepted and ready to land.Jun 16 2022, 10:31 AM
vlorentz updated this revision to Diff 28808.Jun 16 2022, 10:48 AM

add missing case

vlorentz marked an inline comment as done.Jun 16 2022, 10:48 AM
ardumont accepted this revision.Jun 16 2022, 10:49 AM

thx

swh-public-ci added a comment.Jun 16 2022, 10:54 AM

Build is green

Patch application report for D7992 (id=28808)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 62db0cb2086be7a4c6c48f6488fb425485e56093
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/259/ for more details.

Harbormaster completed remote builds in B29893: Diff 28808.Jun 16 2022, 10:54 AM
Closed by commit rDCIDX62db0cb2086b: npm: Add workaround for mangled package descriptions (authored by vlorentz). · Explain WhyJun 16 2022, 4:21 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDX62db0cb2086b: npm: Add workaround for mangled package descriptions.
vlorentz mentioned this in T4277: Deal with null characters in the output of the metadata indexer.Jul 5 2022, 4:51 PM

Revision Contents
Changeset List

PathSize
swh/
indexer/
metadata_dictionary/
58 lines

Diff 28802

View Options

swh/indexer/metadata_dictionary/npm.py

Loading...
About · Developer information · Legal