extract_npm_package_author: Handle list of dict authors layout
ClosedPublic
Actions

Authored by anlambert on Apr 11 2019, 2:50 PM.

Details

Reviewers

olasd

Group Reviewers

Reviewers

Commits

rDLDNPMf6cd8d609448: extract_npm_package_author: Handle list of dict authors layout

Summary

Some package.json files may contain an authors field consisting in
a list of dict. So handle that case to avoid errors such as:

[2019-04-11 12:03:21,650: ERROR/ForkPoolWorker-19] Loading failure, updating to `partial` status
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/loader/core/loader.py", line 893, in load
    more_data_to_fetch = self.fetch_data()
  File "/usr/lib/python3/dist-packages/swh/loader/npm/loader.py", line 203, in fetch_data
    data = next(self.new_versions)
  File "/usr/lib/python3/dist-packages/swh/loader/npm/client.py", line 145, in prepare_package_versions
    version_data)
  File "/usr/lib/python3/dist-packages/swh/loader/npm/client.py", line 200, in _prepare_package_version
    author = extract_npm_package_author(package_json)
  File "/usr/lib/python3/dist-packages/swh/loader/npm/utils.py", line 92, in extract_npm_package_author
    author_data = parse_npm_package_author(package_json['authors'][0])
  File "/usr/lib/python3/dist-packages/swh/loader/npm/utils.py", line 52, in parse_npm_package_author
    author_str.replace('<>', '').replace('()', ''),
AttributeError: 'dict' object has no attribute 'replace'

Related T1644

Diff Detail

Repository

rDLDNPM npm loader

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Apr 11 2019, 2:50 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptApr 11 2019, 2:50 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDNPM/job/tox/5/ for more details.

Harbormaster completed remote builds in B5336: Diff 4527.Apr 11 2019, 2:52 PM

anlambert edited the summary of this revision. (Show Details)Apr 11 2019, 3:10 PM

Update commit message

Build is green
See https://jenkins.softwareheritage.org/job/DLDNPM/job/tox/9/ for more details.

Harbormaster completed remote builds in B5337: Diff 4528.Apr 11 2019, 3:14 PM

This change looks sensible in the context of what's already here, but I'm not sure if favoring the first author over other authors is the best choice here.

I don't remember how we've solved that issue in the case of the deposit, for instance.

Could we turn that question into an issue so we can make a consistent decision across our loaders?

This revision is now accepted and ready to land.Apr 11 2019, 4:11 PM

The deposit loader uses the tar loader under the hood which makes the author or each produced revision to Software Heritage <robot@softwareheritage.org>[1].
The real information about authors can be found in the revision metadata, see [2] as an example.

For npm, a dump of the package.json file is also available in each produced revision metadata, including the full authors list.
I agree that how handling the multiple authors case should be discussed, I have created T1645 on the subject.

[1] https://forge.softwareheritage.org/source/swh-loader-tar/browse/master/swh/loader/tar/build.py$71
[2] https://archive.softwareheritage.org/browse/revision/76b3c170a150af6ee788d799cefe6bf756cabadc/?origin=https://hal.archives-ouvertes.fr/hal-01882337