Page MenuHomeSoftware Heritage
Create Task
Maniphest T2471

NPM package angular-ts-manage fails to be properly loaded
Closed, MigratedEdits Locked
Actions

Description

See https://archive.softwareheritage.org/browse/origin/directory/?origin_url=https://www.npmjs.com/package/angular-ts-manage

no release, no revision, and no full visit.

Most likely related to [1]

[1] https://sentry.softwareheritage.org/organizations/swh/issues/?query=is%3Aunresolved+cannot+be+converted+to+text&statsPeriod=90d

Revisions and Commits

rDLDBASE Generic VCS/Package Loader
AbandonedD3386 Test to reproduce issue
D5423rDLDBASEd84d68a8bb92 package loaders: Stop reading/writing Revision.metadata

Related Objects
Search...

Event Timeline

douardda triaged this task as High priority.Jul 1 2020, 10:05 AM
douardda created this task.
douardda added a comment.EditedJul 1 2020, 10:10 AM

This origin (and others) also fails to be mirrored with errors like:

2020-07-01T08:08:31.034527330Z swh_storage.1.w34ostw2fj07@bespin    | ERROR:root:unsupported Unicode escape sequence
2020-07-01T08:08:31.034547035Z swh_storage.1.w34ostw2fj07@bespin    | DETAIL:  \u0000 cannot be converted to text.
2020-07-01T08:08:31.034550252Z swh_storage.1.w34ostw2fj07@bespin    | CONTEXT:  JSON data, line 1: ...nfig-factory", "version": "0.3.0", "description":...
2020-07-01T08:08:31.034552607Z swh_storage.1.w34ostw2fj07@bespin    | COPY tmp_revision, line 10, column metadata: "{"package_source": {"name": "config-factory", "version": "0.3.0", "filename": "config-factory-0.3.0...."
2020-07-01T08:08:31.034555308Z swh_storage.1.w34ostw2fj07@bespin    | Traceback (most recent call last):
2020-07-01T08:08:31.034557333Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1813, in full_dispatch_request
2020-07-01T08:08:31.034559316Z swh_storage.1.w34ostw2fj07@bespin    |     rv = self.dispatch_request()
2020-07-01T08:08:31.034561208Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/flask/app.py", line 1799, in dispatch_request
2020-07-01T08:08:31.034563201Z swh_storage.1.w34ostw2fj07@bespin    |     return self.view_functions[rule.endpoint](**req.view_args)
2020-07-01T08:08:31.034565055Z swh_storage.1.w34ostw2fj07@bespin    |   File "<decorator-gen-42>", line 2, in revision_add
2020-07-01T08:08:31.034567410Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/core/api/negotiation.py", line 148, in _negotiate
2020-07-01T08:08:31.034569427Z swh_storage.1.w34ostw2fj07@bespin    |     return f.negotiator(*args, **kwargs)
2020-07-01T08:08:31.034571284Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/core/api/negotiation.py", line 82, in __call__
2020-07-01T08:08:31.034573266Z swh_storage.1.w34ostw2fj07@bespin    |     result = self.func(*args, **kwargs)
2020-07-01T08:08:31.034575139Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/core/api/__init__.py", line 453, in _f
2020-07-01T08:08:31.034577163Z swh_storage.1.w34ostw2fj07@bespin    |     return obj_meth(**kw)
2020-07-01T08:08:31.034578968Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/storage/metrics.py", line 24, in d
2020-07-01T08:08:31.034580961Z swh_storage.1.w34ostw2fj07@bespin    |     return f(*a, **kw)
2020-07-01T08:08:31.034582971Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/storage/metrics.py", line 77, in d
2020-07-01T08:08:31.034585007Z swh_storage.1.w34ostw2fj07@bespin    |     r = f(*a, **kw)
2020-07-01T08:08:31.034586849Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/core/db/common.py", line 62, in _meth
2020-07-01T08:08:31.034588741Z swh_storage.1.w34ostw2fj07@bespin    |     return meth(self, *args, db=db, cur=cur, **kwargs)
2020-07-01T08:08:31.034597020Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/storage/storage.py", line 575, in revision_add
2020-07-01T08:08:31.034599212Z swh_storage.1.w34ostw2fj07@bespin    |     lambda rev: parents_filtered.extend(rev["parents"]),
2020-07-01T08:08:31.034601077Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/core/db/__init__.py", line 213, in copy_to
2020-07-01T08:08:31.034602968Z swh_storage.1.w34ostw2fj07@bespin    |     raise exc_info[1].with_traceback(exc_info[2])
2020-07-01T08:08:31.034604780Z swh_storage.1.w34ostw2fj07@bespin    |   File "/usr/lib/python3/dist-packages/swh/core/db/__init__.py", line 175, in writer
2020-07-01T08:08:31.034606696Z swh_storage.1.w34ostw2fj07@bespin    |     "COPY %s (%s) FROM STDIN CSV" % (tblname, ", ".join(columns)), f
2020-07-01T08:08:31.034608754Z swh_storage.1.w34ostw2fj07@bespin    | psycopg2.errors.UntranslatableCharacter: unsupported Unicode escape sequence
2020-07-01T08:08:31.034611057Z swh_storage.1.w34ostw2fj07@bespin    | DETAIL:  \u0000 cannot be converted to text.
2020-07-01T08:08:31.034612976Z swh_storage.1.w34ostw2fj07@bespin    | CONTEXT:  JSON data, line 1: ...nfig-factory", "version": "0.3.0", "description":...
2020-07-01T08:08:31.034615611Z swh_storage.1.w34ostw2fj07@bespin    | COPY tmp_revision, line 10, column metadata: "{"package_source": {"name": "config-factory", "version": "0.3.0", "filename": "config-factory-0.3.0...."
2020-07-01T08:08:31.034618053Z swh_storage.1.w34ostw2fj07@bespin    |
ardumont updated the task description. (Show Details)Jul 1 2020, 10:12 AM
ardumont added a revision: D3386: Test to reproduce issue.Jul 1 2020, 11:25 AM
vlorentz added a subscriber: vlorentz.Aug 31 2020, 4:38 PM

I'm afraid the only way to properly solve this is to wait until we stop writing metadata to the revision table

vlorentz lowered the priority of this task from High to Normal.Mar 15 2021, 12:32 PM
vlorentz added a subtask: T3089: Remove the 'metadata' column of the 'revision' table.
vlorentz added a revision: D5423: package loaders: Stop reading/writing Revision.metadata.Apr 6 2021, 2:18 PM
vlorentz added a commit: rDLDBASEd84d68a8bb92: package loaders: Stop reading/writing Revision.metadata.Apr 6 2021, 3:39 PM
vlorentz claimed this task.Apr 19 2021, 12:43 PM
vlorentz edited subtasks, added: T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table; removed: T3089: Remove the 'metadata' column of the 'revision' table.
olasd added a subscriber: olasd.Apr 23 2021, 1:20 PM

New swh.loader.core deployed in staging.

olasd@worker0:~$ sudo -u swhworker SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_npm.yml swh loader run npm https://www.npmjs.com/package/angular-ts-manage
{'status': 'eventful', 'snapshot_id': 'b34e3cfd3f1aa158241542c92cbd3f92bfd48460'}

https://webapp.staging.swh.network/browse/origin/directory/?origin_url=https://www.npmjs.com/package/angular-ts-manage

Now on to deploying this in production...

ardumont added a subscriber: ardumont.Apr 23 2021, 1:34 PM

Neat :)

olasd closed this task as Resolved.EditedApr 23 2021, 1:41 PM

Now deployed in prod:

olasd@worker01$ sudo -u swhworker SWH_CONFIG_FILENAME=/etc/softwareheritage/loader_npm.yml swh loader run npm https://www.npmjs.com/package/angular-ts-manage
{'status': 'eventful', 'snapshot_id': 'b34e3cfd3f1aa158241542c92cbd3f92bfd48460'}
gitlab-migration changed the task status from Resolved to Migrated.Jan 8 2023, 4:30 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration changed the status of subtask T3138: Make loaders stop writing in the 'metadata' column of the 'revision' table from Resolved to Migrated.Jan 8 2023, 4:33 PM
About · Developer information · Legal