Page MenuHomeSoftware Heritage
Differential D8144

Add NuGet Mapping
ClosedPublic
Actions

Authored by VickyMerzOwn on Jul 20 2022, 7:24 AM.

Details

Reviewers
vlorentz
KShivendu
Group Reviewers
Reviewers
Maniphest Tasks
T4392: Metadata Indexer for NuGet (.nuspec)
Commits
rDCIDX7d7f29fb6a07: metadata_dictionary: Add mappings for "*.nuspec" files
Summary

Closes T4392

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30883
Build 48294: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 48293: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
KShivendu added inline comments.Aug 3 2022, 1:28 PM
swh/indexer/metadata_dictionary/nuget.py
60

I think there's a small misunderstanding here.

he's using strip not split

I also got confused when I first saw it :p

vlorentz added inline comments.Aug 3 2022, 1:46 PM
swh/indexer/metadata_dictionary/nuget.py
60

ah yes, my bad. strip removes all whitespaces by default though; but that's probably a good behavior anyway.

VickyMerzOwn updated this revision to Diff 29518.Aug 3 2022, 2:38 PM
VickyMerzOwn marked 4 inline comments as done.

Use dict instead of Dict
return _translate_dict rather than normalize_dict(_translate_dict)
Use .get("package", {}) instead of .get("package").get("metadata", {})

VickyMerzOwn updated this revision to Diff 29519.Aug 3 2022, 2:42 PM

return _translate_dict rather than normalize_dict(_translate_dict)

swh-public-ci added a comment.Aug 3 2022, 2:46 PM

Build has FAILED

Patch application report for D8144 (id=29518)

Rebasing onto f31f1f5b34...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit e5b39f9fc5ccbf5adb8f2ceeaf631e694f596f31
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/392/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/392/console

Harbormaster failed remote builds in B30585: Diff 29518!Aug 3 2022, 2:46 PM
swh-public-ci added a comment.Aug 3 2022, 2:50 PM

Build has FAILED

Patch application report for D8144 (id=29519)

Rebasing onto f31f1f5b34...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit c88ea6c22e45f41b3c5e5446f87c4403716e89d7
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/393/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/393/console

Harbormaster failed remote builds in B30586: Diff 29519!Aug 3 2022, 2:50 PM
VickyMerzOwn added inline comments.Aug 3 2022, 2:50 PM
swh/indexer/metadata_dictionary/nuget.py
27

oh, yes. Do I use a regex then?

38

I think you mean .get("package").get("metadata", {}) to simplify using .get()'s parameter
Adding the try-except block back. I dropped it because I couldn't think of a test case.

39–41

oh, yeah! makes sense. Updating it..

swh/indexer/tests/metadata_dictionary/test_nuget.py
12

Sorry, didn't understand what you mean. Could you help me? What is a namespace? found some definitions on google but didn't understand what to change here.

vlorentz added inline comments.Aug 3 2022, 2:56 PM
swh/indexer/metadata_dictionary/nuget.py
27

you can't use SingleFileIntrinsicMapping, so you have to reimplement its file-detection method

38

if xmltodict.parse(content.strip(b" \n ")) returns a dict without the package key, then .get("package") returns None, so you call None.get("metadata", {}) which doesn't exist.

swh/indexer/tests/metadata_dictionary/test_nuget.py
12

https://developer.mozilla.org/en-US/docs/Web/SVG/Namespaces_Crash_Course

You replaced the namespace used by nuspec with http://package.example.org/, which changes the meaning of every tag in the file.

VickyMerzOwn updated this revision to Diff 29520.Aug 3 2022, 3:28 PM

Corrects namespace used by nuspec

swh-public-ci added a comment.Aug 3 2022, 3:37 PM

Build has FAILED

Patch application report for D8144 (id=29520)

Rebasing onto f31f1f5b34...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit 1c878121a629cf7c38b15ee106d90ad7a8ddc3a8
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/394/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/394/console

Harbormaster failed remote builds in B30587: Diff 29520!Aug 3 2022, 3:37 PM
VickyMerzOwn updated this revision to Diff 29523.Aug 3 2022, 4:09 PM

Reference dict using square brackets (<dict_name>["<key>"]) instead of <dict_name>.get("<key>") where "<key>" is always in <dict_name>

swh-public-ci added a comment.Aug 3 2022, 4:17 PM

Build has FAILED

Patch application report for D8144 (id=29523)

Rebasing onto f31f1f5b34...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit 68f8f6ba422b129aae030ad761cfb33d4c1d9426
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/395/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/395/console

Harbormaster failed remote builds in B30590: Diff 29523!Aug 3 2022, 4:17 PM
VickyMerzOwn updated this revision to Diff 29713.Aug 12 2022, 10:07 AM

Re-implements file detection mechanism for detecting files with '.nuspec' extension

swh-public-ci added a comment.Aug 12 2022, 10:15 AM

Build has FAILED

Patch application report for D8144 (id=29713)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit d3512f920a13a83bfb6ea042ad381e2845099299
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/415/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/415/console

Harbormaster failed remote builds in B30765: Diff 29713!Aug 12 2022, 10:15 AM
VickyMerzOwn updated this revision to Diff 29725.Aug 12 2022, 4:52 PM

Detects metadata file using string.endswith() instead of regex
Adds test for non-utf8 encoded filenames of Nuget packages

swh-public-ci added a comment.Aug 12 2022, 5:01 PM

Build is green

Patch application report for D8144 (id=29725)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit f41cb2a89a7fdc05200e56293f0cc6b4696c7b45
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/420/ for more details.

Harbormaster completed remote builds in B30776: Diff 29725.Aug 12 2022, 5:01 PM
VickyMerzOwn marked 8 inline comments as done.Aug 15 2022, 1:31 PM
VickyMerzOwn updated this revision to Diff 29728.Aug 15 2022, 1:38 PM

Includes 'readme' field in nuget.csv

VickyMerzOwn marked an inline comment as done.Aug 15 2022, 1:39 PM
VickyMerzOwn marked 7 inline comments as done.Aug 15 2022, 1:41 PM
swh-public-ci added a comment.Aug 15 2022, 1:47 PM

Build is green

Patch application report for D8144 (id=29728)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit c5b47fc44b54b09106c7a3b266f3b828e11e6021
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/421/ for more details.

Harbormaster completed remote builds in B30779: Diff 29728.Aug 15 2022, 1:47 PM
VickyMerzOwn updated this revision to Diff 29771.Aug 17 2022, 6:06 AM

Adds licenseUrl, summary, tags and releaseNotes field to mapping
Adds support for multiple license types

swh-public-ci added a comment.Aug 17 2022, 6:17 AM

Build was aborted

Patch application report for D8144 (id=29771)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 677aeb92b2bbe89a5017eca0e4dbce96e88ecf6e
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/422/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/422/console

Harbormaster failed remote builds in B30823: Diff 29771!Aug 17 2022, 6:17 AM
swh-public-ci added a comment.Aug 17 2022, 9:09 AM

Build is green

Patch application report for D8144 (id=29771)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 1b625d6ccb7b144f7527f0271ba02f9de6f70951
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/423/ for more details.

Harbormaster completed remote builds in B30823: Diff 29771.Aug 17 2022, 9:09 AM
VickyMerzOwn updated this revision to Diff 29800.Aug 18 2022, 11:33 AM

Adds mapping for copyright (*.nuspec) to schema:copyrightNotice property
Checks if local_name is already a canonical name prior to passing through swh.indexer.codemeta.make_absolute_uri()

swh-public-ci added a comment.Aug 18 2022, 11:41 AM

Build is green

Patch application report for D8144 (id=29800)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit c95eca18a6c66d6119379d267b0be068f7287b14
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/424/ for more details.

Harbormaster completed remote builds in B30852: Diff 29800.Aug 18 2022, 11:41 AM
VickyMerzOwn updated this revision to Diff 29831.Aug 18 2022, 7:05 PM

Adds mapping for copyright (*.nuspec) to schema:copyrightNotice property by adding entry to NugetMapping.mapping as suggested by @vlorentz
Excludes the above addition from nuget.csv

swh-public-ci added a comment.Aug 18 2022, 7:13 PM

Build is green

Patch application report for D8144 (id=29831)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit f8a87d3c4fe21942d81bba75835bcee4dc3fbebc
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/425/ for more details.

Harbormaster completed remote builds in B30883: Diff 29831.Aug 18 2022, 7:13 PM
VickyMerzOwn updated this revision to Diff 29833.Aug 18 2022, 9:24 PM

Adds mapping for language (*.nuspec) to schema:inLanguage

swh-public-ci added a comment.Aug 18 2022, 9:33 PM

Build is green

Patch application report for D8144 (id=29833)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 43f96f31a4d53f82c9aa441cb38fae9ef9e3ae5f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/426/ for more details.

Harbormaster completed remote builds in B30884: Diff 29833.Aug 18 2022, 9:33 PM
vlorentz requested changes to this revision.Aug 19 2022, 10:17 AM
vlorentz added inline comments.
swh/indexer/metadata_dictionary/nuget.py
72

remove this statement, the return value is not used.

95–97

This is incorrect as s is not an URL.

Unfortunately, until https://github.com/codemeta/codemeta/pull/280 gets merged, this means we cannot use codemeta's restriction of http://schema.org/releaseNotes, so we have to use http://schema.org/releaseNotes directly.

I'll spare you the back-and-forth, this is how it should be done:

def translate_releaseNotes(self, translated_metadata, s):
    if isinstance(s, str):
        translated_metadata.setdefault("http://schema.org/releaseNotes", []).append(s)

and also, you can remove releaseNotes from string_fields above, it's redundant when either normalize_releaseNotes or translate_releaseNotes is present.

This revision now requires changes to proceed.Aug 19 2022, 10:17 AM
VickyMerzOwn updated this revision to Diff 29836.Aug 19 2022, 10:51 AM

Uses 'http://schema.org/releaseNotes' directly instead of through codemeta's context

swh-public-ci added a comment.Aug 19 2022, 10:59 AM

Build is green

Patch application report for D8144 (id=29836)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 653e0c8f291c1a333306033a37842905489e493d
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/427/ for more details.

Harbormaster completed remote builds in B30887: Diff 29836.Aug 19 2022, 10:59 AM
VickyMerzOwn updated this revision to Diff 29837.Aug 19 2022, 11:00 AM

Excludes redundant return statement in the NuGetMapping.translate_repository() method

swh-public-ci added a comment.Aug 19 2022, 11:08 AM

Build is green

Patch application report for D8144 (id=29837)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 724135cf335fa39989e5012a50773dbe45d02a16
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/428/ for more details.

Harbormaster completed remote builds in B30888: Diff 29837.Aug 19 2022, 11:08 AM
vlorentz added a child revision: D8263: metadata_dictionary: Simplify code using rdflib-style namespace classes.Aug 19 2022, 11:37 AM
VickyMerzOwn updated this revision to Diff 29842.Aug 19 2022, 12:04 PM

Differentiates between supported and unsupported delimiters in NuGetMapping.normalize_license()
Adds support for and delimiter in license expressions

swh-public-ci added a comment.Aug 19 2022, 12:15 PM

Build was aborted

Patch application report for D8144 (id=29842)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit e161c5a82b2ec142e8a6c58db2d800590b51767b
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/430/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/430/console

Harbormaster failed remote builds in B30894: Diff 29842!Aug 19 2022, 12:15 PM
VickyMerzOwn updated this revision to Diff 29845.Aug 19 2022, 12:24 PM

Rebuild

swh-public-ci added a comment.Aug 19 2022, 12:32 PM

Build is green

Patch application report for D8144 (id=29845)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 8b57f9fd8ecf50a95746b20041c2b1c05ad2447b
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/431/ for more details.

Harbormaster completed remote builds in B30897: Diff 29845.Aug 19 2022, 12:32 PM
VickyMerzOwn updated this revision to Diff 29853.Aug 19 2022, 1:57 PM

Adds "AND" to list of unsupported delimiters
Adds word boundaries to unsupported delimiters
Makes unsupported delimiters list case insensitive

swh-public-ci added a comment.Aug 19 2022, 2:05 PM

Build is green

Patch application report for D8144 (id=29853)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 8a2c9b24a190a04da05294fb7bbeea8ebcb9392b
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/432/ for more details.

Harbormaster completed remote builds in B30905: Diff 29853.Aug 19 2022, 2:05 PM
vlorentz accepted this revision.Aug 20 2022, 10:18 AM
vlorentz retitled this revision from Adds NuGet Mappings to Add NuGet Mapping.
vlorentz edited the summary of this revision. (Show Details)
VickyMerzOwn updated this revision to Diff 29881.Aug 20 2022, 11:11 AM

Rebasing

swh-public-ci added a comment.Aug 20 2022, 11:19 AM

Build is green

Patch application report for D8144 (id=29881)

Rebasing onto bb9082a6e5...

Current branch diff-target is up to date.
Changes applied before test
commit 7d7f29fb6a0730ddaaa3b9bc15aef657ba93e5b9
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/433/ for more details.

Harbormaster completed remote builds in B30935: Diff 29881.Aug 20 2022, 11:19 AM
KShivendu accepted this revision.Aug 22 2022, 2:14 PM
This revision is now accepted and ready to land.Aug 22 2022, 2:14 PM
Closed by commit rDCIDX7d7f29fb6a07: metadata_dictionary: Add mappings for "*.nuspec" files (authored by VickyMerzOwn). · Explain WhyAug 22 2022, 2:23 PM
This revision was automatically updated to reflect the committed changes.
VickyMerzOwn added a commit: rDCIDX7d7f29fb6a07: metadata_dictionary: Add mappings for "*.nuspec" files.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
indexer/
data/
68 lines--
metadata_dictionary/
3 lines76%
Loading...
99 lines96%
Loading...
tests/
metadata_dictionary/
150 lines100%
Loading...
1 line100%
Loading...

Diff 29831

View Options

swh/indexer/data/nuget.csv

Loading...
View Options

swh/indexer/metadata_dictionary/__init__.py

Loading...
View Options

swh/indexer/metadata_dictionary/nuget.py

Loading...
View Options

swh/indexer/tests/metadata_dictionary/test_nuget.py

Loading...
View Options

swh/indexer/tests/test_cli.py

Loading...
About · Developer information · Legal