Page MenuHomeSoftware Heritage

Nuget: Lister for NuGet the package manager for .NET
ClosedPublic

Authored by franckbret on Sep 23 2022, 5:51 PM.

Diff Detail

Repository
rDLS Listers
Branch
nuget
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 31758
Build 49689: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 49688: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8528 (id=30731)

Rebasing onto 9b3e565cf7...

Current branch diff-target is up to date.
Changes applied before test
commit 8a9b6a37b476607f52cc7deac6db8a033c09298f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Sep 23 17:39:55 2022 +0200

    [WIP] Nuget: Lister for NuGet the package manager for .NET
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/681/ for more details.

Make use of http_retry instead of throttling_retry decorator after D8519

franckbret retitled this revision from [WIP] Nuget: Lister for NuGet the package manager for .NET to Nuget: Lister for NuGet the package manager for .NET.Sep 26 2022, 12:14 PM

Build is green

Patch application report for D8528 (id=30750)

Rebasing onto d5c30a3ce3...

Current branch diff-target is up to date.
Changes applied before test
commit 895be2af9c284a2d5ddb9bf460d982e56116d141
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Sep 23 17:39:55 2022 +0200

    Nuget: Lister for NuGet the package manager for .NET
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/691/ for more details.

vlorentz added a subscriber: vlorentz.
vlorentz added inline comments.
swh/lister/nuget/lister.py
123–126

What is this API endpoint?

128

Wrong URL. That's probably why no origin is returned in tests.

This revision now requires changes to proceed.Sep 26 2022, 2:01 PM

Make use of generic http_request method after D8520

Build is green

Patch application report for D8528 (id=30773)

Rebasing onto fd1a4244a0...

Current branch diff-target is up to date.
Changes applied before test
commit 1c4031fcda84b937f7bcff30d90ec93585cdfa67
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Sep 23 17:39:55 2022 +0200

    Nuget: Lister for NuGet the package manager for .NET
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/695/ for more details.

More complete tests and basic documentation

Add .nuspec fixture that contains 'repository' entry which is used as origin url
Add basic documentation about how the lister work

Build is green

Patch application report for D8528 (id=30781)

Rebasing onto fd1a4244a0...

Current branch diff-target is up to date.
Changes applied before test
commit 50d1293fae37a4aa99c072db38b9d7f6e0f8afc8
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Sep 23 17:39:55 2022 +0200

    Nuget: Lister for NuGet the package manager for .NET
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/697/ for more details.

franckbret added a subscriber: anlambert.
franckbret added inline comments.
swh/lister/nuget/lister.py
123–126

The api endpoint to download package manifest
https://learn.microsoft.com/en-us/nuget/api/package-base-address-resource#download-package-manifest-nuspec

When looking at it with @anlambert we discover that the linked archives did contains binary, not real source code.
Getting the nuspec package information is the only way to get a dvcs information. "repository" information is sometimes here, sometimes empty.

I've send an email to support to be sure, no answer for now.

128

nice catch, it wasn't that directly that end to have origins in tests, but the fact that those ones does not have a 'repository' entry in their nuspec file.
Last patch add such fixture.

Overall looks good to me, just a couple of inline comments to handle.

swh/lister/nuget/__init__.py
52

Use compose v2 command instead: docker compose up -d

54–60
Then schedule a nuget listing task::

   docker compose exec swh-scheduler swh scheduler task add -p oneshot list-nuget

You can follow lister execution by displaying logs of swh-lister service::

   docker compose logs -f swh-lister
swh/lister/nuget/lister.py
23–26

You can remove the user-agent setting code, it is now handled in base lister class.

33

Could be svn or hg too, better not setting a visit type here.

62

You can remove the params parameter here.

69

ditto

93

ditto

108

ditto

115–116

You can merge this two lines into a single one.

swh/lister/nuget/tests/test_lister.py
4

Please add a new line between license header and imports.

This revision now requires changes to proceed.Sep 26 2022, 4:29 PM
franckbret marked 12 inline comments as done.

Update docker usage documentation section and remove some useless code

Build is green

Patch application report for D8528 (id=30784)

Rebasing onto fd1a4244a0...

Current branch diff-target is up to date.
Changes applied before test
commit db3633769b9d5be4d72763163a4f9492c4b41962
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Sep 23 17:39:55 2022 +0200

    Nuget: Lister for NuGet the package manager for .NET
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/699/ for more details.

swh/lister/nuget/__init__.py
11–13

now that I understand the lister better; could you mention it prominently here that the lister discovers origins on other forges because NuGet packages are binaries?

(I know "Origins from page" says it already, but it's pretty important)

Explain that the lister discovers origins on other forges because NuGet packages are binaries

Build is green

Patch application report for D8528 (id=30831)

Rebasing onto cd596eb2b4...

Current branch diff-target is up to date.
Changes applied before test
commit 3928fc9ee9ff98497b3a2e7151c72f42850fe799
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Sep 23 17:39:55 2022 +0200

    Nuget: Lister for NuGet the package manager for .NET
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/710/ for more details.

Build is green

Patch application report for D8528 (id=30847)

Rebasing onto cd596eb2b4...

Current branch diff-target is up to date.
Changes applied before test
commit 3928fc9ee9ff98497b3a2e7151c72f42850fe799
Author: Franck Bret <franck.bret@octobus.net>
Date:   Fri Sep 23 17:39:55 2022 +0200

    Nuget: Lister for NuGet the package manager for .NET
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/712/ for more details.

This revision is now accepted and ready to land.Sep 27 2022, 4:22 PM