Page MenuHomeSoftware Heritage

Add NuGet Mapping
ClosedPublic

Authored by VickyMerzOwn on Jul 20 2022, 7:24 AM.

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 30585
Build 47817: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 47816: arc lint + arc unit

Unit TestsFailed

TimeTest
947 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.storage.test_api_client.TestIndexerStorageContentMetadata::test_add_deadlock
self = <contextlib.ExitStack object at 0x7f088276bcc0> exc_details = (<class 'ValueError'>, ValueError("<Token var=<ContextVar name='flask.request_ctx' at 0x7f08955e7f10> at 0x7f0882425750> was created in a different Context"), <traceback object at 0x7f0882428c48>) received_exc = False
111 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.storage.test_api_client.TestIndexerStorageContentMetadata::test_add_deadlock
def teardown(): > ctx.pop()
833 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.storage.test_api_client.TestIndexerStorageContentMimetypes::test_add_deadlock
self = <contextlib.ExitStack object at 0x7f0882638f98> exc_details = (<class 'ValueError'>, ValueError("<Token var=<ContextVar name='flask.request_ctx' at 0x7f08955e7f10> at 0x7f08824aff78> was created in a different Context"), <traceback object at 0x7f08824b46c8>) received_exc = False
108 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.storage.test_api_client.TestIndexerStorageContentMimetypes::test_add_deadlock
def teardown(): > ctx.pop()
972 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.tests.storage.test_api_client.TestIndexerStorageDirectoryIntrinsicMetadata::test_add_deadlock
self = <contextlib.ExitStack object at 0x7f0882429470> exc_details = (<class 'ValueError'>, ValueError("<Token var=<ContextVar name='flask.request_ctx' at 0x7f08955e7f10> at 0x7f08822d5e58> was created in a different Context"), <traceback object at 0x7f08822f1788>) received_exc = False
View Full Test Results (10 Failed · 326 Passed · 9 Skipped)

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
swh/indexer/metadata_dictionary/nuget.py
60

I think there's a small misunderstanding here.

he's using strip not split

I also got confused when I first saw it :p

swh/indexer/metadata_dictionary/nuget.py
60

ah yes, my bad. strip removes all whitespaces by default though; but that's probably a good behavior anyway.

VickyMerzOwn marked 4 inline comments as done.

Use dict instead of Dict
return _translate_dict rather than normalize_dict(_translate_dict)
Use .get("package", {}) instead of .get("package").get("metadata", {})

return _translate_dict rather than normalize_dict(_translate_dict)

Build has FAILED

Patch application report for D8144 (id=29518)

Rebasing onto f31f1f5b34...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit e5b39f9fc5ccbf5adb8f2ceeaf631e694f596f31
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/392/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/392/console

Build has FAILED

Patch application report for D8144 (id=29519)

Rebasing onto f31f1f5b34...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit c88ea6c22e45f41b3c5e5446f87c4403716e89d7
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/393/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/393/console

swh/indexer/metadata_dictionary/nuget.py
27

oh, yes. Do I use a regex then?

38

I think you mean .get("package").get("metadata", {}) to simplify using .get()'s parameter
Adding the try-except block back. I dropped it because I couldn't think of a test case.

39–41

oh, yeah! makes sense. Updating it..

swh/indexer/tests/metadata_dictionary/test_nuget.py
12

Sorry, didn't understand what you mean. Could you help me? What is a namespace? found some definitions on google but didn't understand what to change here.

swh/indexer/metadata_dictionary/nuget.py
27

you can't use SingleFileIntrinsicMapping, so you have to reimplement its file-detection method

38

if xmltodict.parse(content.strip(b" \n ")) returns a dict without the package key, then .get("package") returns None, so you call None.get("metadata", {}) which doesn't exist.

swh/indexer/tests/metadata_dictionary/test_nuget.py
12

https://developer.mozilla.org/en-US/docs/Web/SVG/Namespaces_Crash_Course

You replaced the namespace used by nuspec with http://package.example.org/, which changes the meaning of every tag in the file.

Corrects namespace used by nuspec

Build has FAILED

Patch application report for D8144 (id=29520)

Rebasing onto f31f1f5b34...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit 1c878121a629cf7c38b15ee106d90ad7a8ddc3a8
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/394/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/394/console

Reference dict using square brackets (<dict_name>["<key>"]) instead of <dict_name>.get("<key>") where "<key>" is always in <dict_name>

Build has FAILED

Patch application report for D8144 (id=29523)

Rebasing onto f31f1f5b34...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit 68f8f6ba422b129aae030ad761cfb33d4c1d9426
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/395/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/395/console

Re-implements file detection mechanism for detecting files with '.nuspec' extension

Build has FAILED

Patch application report for D8144 (id=29713)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for .nuspec
Changes applied before test
commit d3512f920a13a83bfb6ea042ad381e2845099299
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for .nuspec

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/415/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/415/console

Detects metadata file using string.endswith() instead of regex
Adds test for non-utf8 encoded filenames of Nuget packages

Build is green

Patch application report for D8144 (id=29725)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit f41cb2a89a7fdc05200e56293f0cc6b4696c7b45
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/420/ for more details.

Includes 'readme' field in nuget.csv

Build is green

Patch application report for D8144 (id=29728)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit c5b47fc44b54b09106c7a3b266f3b828e11e6021
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/421/ for more details.

Adds licenseUrl, summary, tags and releaseNotes field to mapping
Adds support for multiple license types

Build was aborted

Patch application report for D8144 (id=29771)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 677aeb92b2bbe89a5017eca0e4dbce96e88ecf6e
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/422/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/422/console

Build is green

Patch application report for D8144 (id=29771)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 1b625d6ccb7b144f7527f0271ba02f9de6f70951
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/423/ for more details.

Adds mapping for copyright (*.nuspec) to schema:copyrightNotice property
Checks if local_name is already a canonical name prior to passing through swh.indexer.codemeta.make_absolute_uri()

Build is green

Patch application report for D8144 (id=29800)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit c95eca18a6c66d6119379d267b0be068f7287b14
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/424/ for more details.

Adds mapping for copyright (*.nuspec) to schema:copyrightNotice property by adding entry to NugetMapping.mapping as suggested by @vlorentz
Excludes the above addition from nuget.csv

Build is green

Patch application report for D8144 (id=29831)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit f8a87d3c4fe21942d81bba75835bcee4dc3fbebc
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/425/ for more details.

Adds mapping for language (*.nuspec) to schema:inLanguage

Build is green

Patch application report for D8144 (id=29833)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 43f96f31a4d53f82c9aa441cb38fae9ef9e3ae5f
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/426/ for more details.

vlorentz added inline comments.
swh/indexer/metadata_dictionary/nuget.py
72

remove this statement, the return value is not used.

95–97

This is incorrect as s is not an URL.

Unfortunately, until https://github.com/codemeta/codemeta/pull/280 gets merged, this means we cannot use codemeta's restriction of http://schema.org/releaseNotes, so we have to use http://schema.org/releaseNotes directly.

I'll spare you the back-and-forth, this is how it should be done:

def translate_releaseNotes(self, translated_metadata, s):
    if isinstance(s, str):
        translated_metadata.setdefault("http://schema.org/releaseNotes", []).append(s)

and also, you can remove releaseNotes from string_fields above, it's redundant when either normalize_releaseNotes or translate_releaseNotes is present.

This revision now requires changes to proceed.Aug 19 2022, 10:17 AM

Uses 'http://schema.org/releaseNotes' directly instead of through codemeta's context

Build is green

Patch application report for D8144 (id=29836)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 653e0c8f291c1a333306033a37842905489e493d
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/427/ for more details.

Excludes redundant return statement in the NuGetMapping.translate_repository() method

Build is green

Patch application report for D8144 (id=29837)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 724135cf335fa39989e5012a50773dbe45d02a16
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/428/ for more details.

Differentiates between supported and unsupported delimiters in NuGetMapping.normalize_license()
Adds support for and delimiter in license expressions

Build was aborted

Patch application report for D8144 (id=29842)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit e161c5a82b2ec142e8a6c58db2d800590b51767b
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/430/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/430/console

Build is green

Patch application report for D8144 (id=29845)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 8b57f9fd8ecf50a95746b20041c2b1c05ad2447b
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/431/ for more details.

Adds "AND" to list of unsupported delimiters
Adds word boundaries to unsupported delimiters
Makes unsupported delimiters list case insensitive

Build is green

Patch application report for D8144 (id=29853)

Rebasing onto bb9082a6e5...

First, rewinding head to replay your work on top of it...
Applying: metadata_dictionary: Add mappings for "*.nuspec" files
Changes applied before test
commit 8a2c9b24a190a04da05294fb7bbeea8ebcb9392b
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/432/ for more details.

vlorentz retitled this revision from Adds NuGet Mappings to Add NuGet Mapping.
vlorentz edited the summary of this revision. (Show Details)

Build is green

Patch application report for D8144 (id=29881)

Rebasing onto bb9082a6e5...

Current branch diff-target is up to date.
Changes applied before test
commit 7d7f29fb6a0730ddaaa3b9bc15aef657ba93e5b9
Author: Satvik Vemuganti <vemugantisesha@iitbhilai.ac.in>
Date:   Wed Jul 20 10:52:34 2022 +0530

    metadata_dictionary: Add mappings for "*.nuspec" files

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/433/ for more details.

This revision is now accepted and ready to land.Aug 22 2022, 2:14 PM