storage: Strip null characters from metadata documents
ClosedPublic
Actions

Authored by vlorentz on Aug 10 2022, 10:45 AM.

Details

Reviewers

anlambert

Group Reviewers

Reviewers

Maniphest Tasks

T4277: Deal with null characters in the output of the metadata indexer

Commits

rDCIDXbb9082a6e5b9: storage: Strip null characters from metadata documents

Summary

They cause postgresql to crash because it does not allow them in text fields.

They are seemingly only present accidentally in source documents;
so stripping them does not really impact the quality of metadata.

Resolves T4277.

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 30738
Build 48059: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 48058: arc lint + arc unit

Event Timeline

vlorentz created this revision.Aug 10 2022, 10:45 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptAug 10 2022, 10:45 AM

Build is green

Patch application report for D8229 (id=29685)

Rebasing onto 5313be86b3...

Current branch diff-target is up to date.

Changes applied before test

commit bb9082a6e5b95085ada61a917a2547f7d0a5c5e2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 10 10:44:31 2022 +0200

    storage: Strip null characters from metadata documents
    
    They cause postgresql to crash because it does not allow them in text fields.
    
    They are seemingly only present accidentally in source documents;
    so stripping them does not really impact the quality of metadata.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/413/ for more details.

Harbormaster completed remote builds in B30738: Diff 29685.Aug 10 2022, 10:54 AM

vlorentz requested review of this revision.Aug 10 2022, 10:54 AM

Looks good to me.

swh/indexer/storage/__init__.py
53	s/NUL/NULL/

This revision is now accepted and ready to land.Aug 10 2022, 11:52 AM

vlorentz added inline comments.Aug 10 2022, 11:59 AM

swh/indexer/storage/__init__.py
53	nah, `NUL` is the name of the zero byte/character in ASCII, `NULL` is the name inherited from C for zero pointers. Unicode doesn't have a name for the zero character, so I used ASCII's