Page MenuHomeSoftware Heritage
Create Task
Maniphest T3201

Mirror: unsupported Unicode escape sequence
Closed, MigratedEdits Locked
Actions

Description

2021-04-01T23:58:53.195097589Z swh_storage.1.yjtfpr7ambpm@libra    | ERROR:root:unsupported Unicode escape sequence
2021-04-01T23:58:53.195111929Z swh_storage.1.yjtfpr7ambpm@libra    | DETAIL:  \u0000 cannot be converted to text.
2021-04-01T23:58:53.195115416Z swh_storage.1.yjtfpr7ambpm@libra    | CONTEXT:  JSON data, line 1: .../cookbook-ui", "version": "0.0.8", "description":...
2021-04-01T23:58:53.195118594Z swh_storage.1.yjtfpr7ambpm@libra    | COPY tmp_revision, line 135, column metadata: "{"intrinsic": {"tool": "package.json", "raw": {"name": "@caigua/cookbook-ui", "version": "0.0.8", "d..."
2021-04-01T23:58:53.195121602Z swh_storage.1.yjtfpr7ambpm@libra    | Traceback (most recent call last):
2021-04-01T23:58:53.195123371Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1813, in full_dispatch_request
2021-04-01T23:58:53.195125245Z swh_storage.1.yjtfpr7ambpm@libra    |     rv = self.dispatch_request()
2021-04-01T23:58:53.195126966Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1799, in dispatch_request
2021-04-01T23:58:53.195129345Z swh_storage.1.yjtfpr7ambpm@libra    |     return self.view_functions[rule.endpoint](**req.view_args)
2021-04-01T23:58:53.195132261Z swh_storage.1.yjtfpr7ambpm@libra    |   File "<decorator-gen-40>", line 2, in revision_add
2021-04-01T23:58:53.195134643Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/core/api/negotiation.py", line 147, in _negotiate
2021-04-01T23:58:53.195137012Z swh_storage.1.yjtfpr7ambpm@libra    |     return f.negotiator(*args, **kwargs)
2021-04-01T23:58:53.195139267Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/core/api/negotiation.py", line 81, in __call__
2021-04-01T23:58:53.195141571Z swh_storage.1.yjtfpr7ambpm@libra    |     result = self.func(*args, **kwargs)
2021-04-01T23:58:53.195143939Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 458, in _f
2021-04-01T23:58:53.195146511Z swh_storage.1.yjtfpr7ambpm@libra    |     return obj_meth(**kw)
2021-04-01T23:58:53.195149288Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/storage/metrics.py", line 24, in d
2021-04-01T23:58:53.195151859Z swh_storage.1.yjtfpr7ambpm@libra    |     return f(*a, **kw)
2021-04-01T23:58:53.195154111Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/storage/metrics.py", line 77, in d
2021-04-01T23:58:53.195156797Z swh_storage.1.yjtfpr7ambpm@libra    |     r = f(*a, **kw)
2021-04-01T23:58:53.195159094Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
2021-04-01T23:58:53.195168730Z swh_storage.1.yjtfpr7ambpm@libra    |     return meth(self, *args, db=db, cur=cur, **kwargs)
2021-04-01T23:58:53.195171362Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/storage/postgresql/storage.py", line 581, in revision_add
2021-04-01T23:58:53.195173928Z swh_storage.1.yjtfpr7ambpm@libra    |     lambda rev: parents_filtered.extend(rev["parents"]),
2021-04-01T23:58:53.195176318Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/core/db/__init__.py", line 294, in copy_to
2021-04-01T23:58:53.195178855Z swh_storage.1.yjtfpr7ambpm@libra    |     raise exc_info[1].with_traceback(exc_info[2])
2021-04-01T23:58:53.195180970Z swh_storage.1.yjtfpr7ambpm@libra    |   File "/usr/lib/python3/dist-packages/swh/core/db/__init__.py", line 244, in writer
2021-04-01T23:58:53.195183038Z swh_storage.1.yjtfpr7ambpm@libra    |     "COPY %s (%s) FROM STDIN" % (tblname, ", ".join(columns)), f
2021-04-01T23:58:53.195185541Z swh_storage.1.yjtfpr7ambpm@libra    | psycopg2.errors.UntranslatableCharacter: unsupported Unicode escape sequence
2021-04-01T23:58:53.195188056Z swh_storage.1.yjtfpr7ambpm@libra    | DETAIL:  \u0000 cannot be converted to text.
2021-04-01T23:58:53.195189718Z swh_storage.1.yjtfpr7ambpm@libra    | CONTEXT:  JSON data, line 1: .../cookbook-ui", "version": "0.0.8", "description":...
2021-04-01T23:58:53.195191996Z swh_storage.1.yjtfpr7ambpm@libra    | COPY tmp_revision, line 135, column metadata: "{"intrinsic": {"tool": "package.json", "raw": {"name": "@caigua/cookbook-ui", "version": "0.0.8", "d..."

context;

ii  python3-swh.core        0.12.0-1~swh1~bpo10+1 all          Software Heritage core utilities
ii  python3-swh.journal     0.7.1-1~swh1~bpo10+1  all          Software Heritage Journal utilities
ii  python3-swh.model       2.3.0-1~swh1~bpo10+1  all          Software Heritage data model
ii  python3-swh.objstorage  0.2.2-1~swh1~bpo10+1  all          Software Heritage Object Storage
ii  python3-swh.scheduler   0.10.0-1~swh1~bpo10+1 all          Software Heritage Scheduler
ii  python3-swh.storage     0.27.0-1~swh1~bpo10+1 all          Software Heritage storage utilities

Revisions and Commits

rDSTO Storage manager
D5414rDSTO39507b24d0f4 Make the replayer drop the Revision.metadata

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT3197 Mirror: fix common issues of a replayer session
 Migratedgitlab-migrationT3201 Mirror: unsupported Unicode escape sequence
 Migratedgitlab-migrationT3089 Remove the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT3090 Make loaders not rely on the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT2849 Design and implement a mapping from "original VCS ids" to SWHIDs to help incremental loaders
 Migratedgitlab-migrationT3138 Make loaders stop writing in the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT3139 Make loaders stop reading from the 'metadata' column of the 'revision' table
 Migratedgitlab-migrationT3140 Make loaders use ExtId to find what artifacts are already loaded, in addition to revision.metadata
 Migratedgitlab-migrationT3143 Migrate revision metadata to extid in the storage
 Migratedgitlab-migrationT3142 Make loaders write to the ExtId storage
 Migratedgitlab-migrationT3141 Make loaders pass ExtId objects between functions instead of "known_artifact"
 Migratedgitlab-migrationT2564 migrate existing revisions metadata extra_headers to actual extra_headers field
 Migratedgitlab-migrationT2513 Copy metadata on revisions to the extrinsic metadata storage
 Migratedgitlab-migrationT2512 Make all loaders write their extrinsic metadata to the appropriate storage
 Migratedgitlab-migrationT2496 Write deposit metadata for revisions in the generic metadata storage
 Migratedgitlab-migrationT2306 Generic storage for extrinsic, qualified metadata related to any node of the swh archive
 Migratedgitlab-migrationT2703 Use intrinsic identifiers/hashes for RawExtrinsicMetadata objects
 Migratedgitlab-migrationT3017 Use hashes as keys in swh.journal.objects.raw_extrinsic_metadata
 Migratedgitlab-migrationT3018 Allow querying raw_extrinsic_metadata by hash in swh-storage
 Migratedgitlab-migrationT3022 Deduplicate RawExtrinsicMetadata by hash instead of a subset of their fields
 Migratedgitlab-migrationT3019 Add an index for raw_extrinsic_metadata.id in swh.storage.postgresql
 Migratedgitlab-migrationT3020 Add an "index" for raw_extrinsic_metadata.id in swh.storage.cassandra
 Migratedgitlab-migrationT3074 Migrate all packages away from the old SWHID class
 Migratedgitlab-migrationT3034 generalize usage of SWHID for referencing SWH archive objects

Event Timeline

douardda triaged this task as High priority.Apr 2 2021, 9:54 AM
douardda created this task.
douardda added a comment.Apr 2 2021, 9:59 AM

from #swh-devel

09:45 <+vlorentz> douardda: yes and the only way around it (short of dropping data) is T3089
09:46 -swhbot:#swh-devel- T3089 (submitter: vlorentz, owner: vlorentz, status: Open): Remove the 'metadata' column of the 'revision' table <https://forge.softwareheritage.org/T3089>
09:46 <+vlorentz> or switching to cassandra
09:46 <+vlorentz> the good news is, they couldn't be inserted in the storage either, so you can safely drop them for now
douardda added a comment.Apr 2 2021, 10:05 AM

easy fix: modify the replayer to ignore this 'metadata' column while inserting revisions

vlorentz added a subtask: T3089: Remove the 'metadata' column of the 'revision' table.Apr 6 2021, 2:20 PM
vlorentz added a revision: D5414: Make the replayer drop the Revision.metadata.Apr 6 2021, 4:25 PM
douardda closed this task as Resolved by committing rDSTO39507b24d0f4: Make the replayer drop the Revision.metadata.Apr 6 2021, 4:42 PM
douardda claimed this task.
douardda added a commit: rDSTO39507b24d0f4: Make the replayer drop the Revision.metadata.
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 10:02 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration closed subtask T3089: Remove the 'metadata' column of the 'revision' table as Migrated.Jan 8 2023, 10:22 PM
About · Developer information · Legal