Page MenuHomeSoftware Heritage
Differential D7992

npm: Add workaround for mangled package descriptions
ClosedPublic
Actions

Authored by vlorentz on Jun 15 2022, 6:30 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T4277: Deal with null characters in the output of the metadata indexer
Commits
rDCIDX62db0cb2086b: npm: Add workaround for mangled package descriptions
Summary

Null bytes in JSON produced by indexers cause the indexer-storage to crash (T4277),
and this case seems to be the only current source of such crashes;
so this should fix the issue for now.

A future commit will sanitize all JSON documents before storage.

Diff Detail

Repository
rDCIDX Metadata indexer
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Jun 15 2022, 6:30 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 15 2022, 6:30 PM
vlorentz added a task: T4277: Deal with null characters in the output of the metadata indexer.Jun 15 2022, 6:30 PM
swh-public-ci added a comment.Jun 15 2022, 6:35 PM

Build is green

Patch application report for D7992 (id=28796)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 026e9fb86c138df9f62473a0a03cf1cf9fd1617b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/255/ for more details.

Harbormaster completed remote builds in B29878: Diff 28796.Jun 15 2022, 6:35 PM
vlorentz requested review of this revision.Jun 15 2022, 6:35 PM
vlorentz updated this revision to Diff 28802.Jun 16 2022, 9:58 AM

fix outdated comment

swh-public-ci added a comment.Jun 16 2022, 10:04 AM

Build has FAILED

Patch application report for D7992 (id=28802)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 92fc18eae4c239e0c5811060d706986ea54063f7
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/256/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/256/console

Harbormaster failed remote builds in B29885: Diff 28802!Jun 16 2022, 10:04 AM
vlorentz updated this revision to Diff 28803.Jun 16 2022, 10:17 AM

fix None support

vlorentz updated this revision to Diff 28804.Jun 16 2022, 10:21 AM

fix test

swh-public-ci added a comment.Jun 16 2022, 10:23 AM

Build has FAILED

Patch application report for D7992 (id=28803)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 4c1a829a6b3331c89a0af1c1234b58e271c25b52
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/257/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/257/console

Harbormaster failed remote builds in B29886: Diff 28803!Jun 16 2022, 10:23 AM
swh-public-ci added a comment.Jun 16 2022, 10:26 AM

Build is green

Patch application report for D7992 (id=28804)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 38bd1f45cda2599730821b07d446c9c37f0d8af1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/258/ for more details.

Harbormaster completed remote builds in B29887: Diff 28804.Jun 16 2022, 10:26 AM
ardumont accepted this revision.Jun 16 2022, 10:31 AM
ardumont added a subscriber: ardumont.

lgtm, one remark inline.

swh/indexer/metadata_dictionary/npm.py
183

you're missing that one in your docstring sample tests.

This revision is now accepted and ready to land.Jun 16 2022, 10:31 AM
vlorentz updated this revision to Diff 28808.Jun 16 2022, 10:48 AM

add missing case

vlorentz marked an inline comment as done.Jun 16 2022, 10:48 AM
ardumont accepted this revision.Jun 16 2022, 10:49 AM

thx

swh-public-ci added a comment.Jun 16 2022, 10:54 AM

Build is green

Patch application report for D7992 (id=28808)

Rebasing onto 710467138a...

Current branch diff-target is up to date.
Changes applied before test
commit 62db0cb2086be7a4c6c48f6488fb425485e56093
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Jun 15 18:29:09 2022 +0200

    npm: Add workaround for mangled package descriptions
    
    Null bytes in JSON produced by indexers cause the indexer-storage to crash,
    and this case seems to be the only current source of such crashes;
    so this should fix the issue for now.
    
    A future commit will sanitize all JSON documents before storage.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/259/ for more details.

Harbormaster completed remote builds in B29893: Diff 28808.Jun 16 2022, 10:54 AM
Closed by commit rDCIDX62db0cb2086b: npm: Add workaround for mangled package descriptions (authored by vlorentz). · Explain WhyJun 16 2022, 4:21 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDX62db0cb2086b: npm: Add workaround for mangled package descriptions.
vlorentz mentioned this in T4277: Deal with null characters in the output of the metadata indexer.Jul 5 2022, 4:51 PM

Revision Contents
Changeset List

PathSize
swh/
indexer/
metadata_dictionary/
67 lines

Diff 28825

View Options

swh/indexer/metadata_dictionary/npm.py

Loading...
About · Developer information · Legal