Maniphest T4277

Deal with null characters in the output of the metadata indexer
Closed, MigratedEdits Locked
Actions

Assigned To

gitlab-migration

Authored By

	vlorentz
	May 24 2022, 5:42 PM

Tags

Subscribers

gitlab-migration

swh-sentry-integration

Description

They cause the indexer to crash because postgresql does not allow null characters in text fields

https://sentry.softwareheritage.org/share/issue/fe0e83d761524a32b72de59985454a25/

Revisions and Commits

rDCIDX Metadata indexer
	D8229	rDCIDXbb9082a6e5b9 storage: Strip null characters from metadata documents
	D7992	rDCIDX62db0cb2086b npm: Add workaround for mangled package descriptions

Related Objects
Search...

		Status	Assigned	Task
		Migrated	gitlab-migration	T4392 Metadata Indexer for NuGet (.nuspec)
		Migrated	gitlab-migration	T3097 Expose metadata in the WebApp and make it searchable
		Migrated	gitlab-migration	T2064 Add metadata from deposits to metadata search
		Migrated	gitlab-migration	T2073 Index extrinsic metadata from the journal in swh-search/Elasticsearch
		Migrated	gitlab-migration	T4273 Rewrite indexers as journal clients when relevant
		Migrated	gitlab-migration	T4282 Deploy new origin intrinsic metadata journal client indexer > v1.1
		Migrated	gitlab-migration	T4274 Resolve all known crashes in the metadata indexer
		Migrated	gitlab-migration	T4277 Deal with null characters in the output of the metadata indexer
				Restricted Maniphest Task
		Migrated	gitlab-migration	T4401 Index metadata from the deposit
		Migrated	gitlab-migration	T4459 Deploy swh-indexer > v2.6 on staging then production
		Migrated	gitlab-migration	T4429 Deploy swh-indexer v2.3.0 on production and staging
		Migrated	gitlab-migration	T4477 staging origin intrinsic metadata indexer are stuck
		Migrated	gitlab-migration	T4606 Deploy swh-indexer v2.7.0
		Migrated	gitlab-migration	T4694 Use directory metadata in origin search

Event Timeline

vlorentz triaged this task as Normal priority.May 24 2022, 5:42 PM

vlorentz created this task.

vlorentz mentioned this in D7992: npm: Add workaround for mangled package descriptions.Jun 15 2022, 6:30 PM

vlorentz added a revision: D7992: npm: Add workaround for mangled package descriptions.Jun 15 2022, 6:30 PM

vlorentz added a commit: rDCIDX62db0cb2086b: npm: Add workaround for mangled package descriptions.Jun 16 2022, 4:21 PM

Sentry issue: SWH-INDEXER-FB

lowering priority, as it is still possible for the crash to happen, but D7992 solves the only known source

bumping it up, because I found another source: https://github.com/james-tracy02/pixel-online/blob/master/package.json#L4

vlorentz added a revision: D8229: storage: Strip null characters from metadata documents.Aug 10 2022, 10:45 AM

vlorentz added a commit: rDCIDXbb9082a6e5b9: storage: Strip null characters from metadata documents.Aug 10 2022, 12:06 PM

vlorentz added a subtask: T4429: Deploy swh-indexer v2.3.0 on production and staging.Aug 10 2022, 12:15 PM

ardumont closed subtask T4429: Deploy swh-indexer v2.3.0 on production and staging as Resolved.Aug 30 2022, 10:01 AM

vlorentz closed this task as Resolved.Aug 30 2022, 10:04 AM

gitlab-migration changed the status of subtask T4429: Deploy swh-indexer v2.3.0 on production and staging from Resolved to Migrated.Oct 19 2022, 6:07 PM

This task has been migrated to GitLab.