Page MenuHomeSoftware Heritage
Differential D8748

Nuget: Implement incremental listing
ClosedPublic
Actions

Authored by franckbret on Oct 19 2022, 4:15 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDLSea146ce297d5: Nuget: Implement incremental listing
Summary

The lister is incremental and based on the value of `commitTimeStamp` retrieved on index http api endpoint.

Related T1718

Diff Detail

Repository
rDLS Listers
Branch
nuget-incremental
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32520
Build 50931: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 50930: arc lint + arc unit

Event Timeline

franckbret created this revision.Oct 19 2022, 4:15 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 19 2022, 4:15 PM
swh-public-ci added a comment.Oct 19 2022, 4:21 PM

Build is green

Patch application report for D8748 (id=31528)

Rebasing onto 0baaf68cff...

First, rewinding head to replay your work on top of it...
Applying: Nuget: Implement incremental listing
Changes applied before test
commit f67d9e2ffcee59725ee3e8a66bc6673fc7dc560e
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 19 16:11:23 2022 +0200

    Nuget: Implement incremental listing
    
    The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/804/ for more details.

Harbormaster completed remote builds in B32482: Diff 31528.Oct 19 2022, 4:21 PM
franckbret requested review of this revision.Oct 19 2022, 4:21 PM
douardda added a subscriber: douardda.Oct 20 2022, 10:39 AM

it's unclear to me whether this actually implements the iteration protocol described in the API doc (https://learn.microsoft.com/en-us/nuget/api/catalog-resource#iterating-over-catalog-items) or not.

douardda added a comment.Oct 20 2022, 10:41 AM
In D8748#227318, @douardda wrote:

it's unclear to me whether this actually implements the iteration protocol described in the API doc (https://learn.microsoft.com/en-us/nuget/api/catalog-resource#iterating-over-catalog-items) or not.

To be clear, I think it should and it should be mentioned somewhere in a comment/docstring that this is what the code does, with a link to this precise part of the API documentation.

franckbret updated this revision to Diff 31566.Oct 24 2022, 11:48 AM

Improve documentation section related to incremental listing

swh-public-ci added a comment.Oct 24 2022, 11:54 AM

Build is green

Patch application report for D8748 (id=31566)

Rebasing onto 8a82bbf95f...

First, rewinding head to replay your work on top of it...
Applying: Nuget: Implement incremental listing
Changes applied before test
commit 7b67c70c37a33ec7c36b1395b2049ff588e0a69f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 19 16:11:23 2022 +0200

    Nuget: Implement incremental listing
    
    The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/810/ for more details.

Harbormaster completed remote builds in B32520: Diff 31566.Oct 24 2022, 11:54 AM
vlorentz added a subscriber: vlorentz.Nov 4 2022, 9:27 AM

Nice to have iteration documented officially by the API for once :)

Looks good overall, but some minor comments below, as usual

swh/lister/nuget/__init__.py
72–77
swh/lister/nuget/lister.py
79

unless it can be absent (in which case you need to do something with it instead of passing it to iso8601.parse_date)

89–91

(not A) or (A and B) is equivalent to (not A) or B

147–148

it is possible for commitTimeStamp to be missing? if yes, please add a test for it

swh/lister/nuget/tests/test_lister.py
50

remove this assertion, it's redundant with the next one, and gives less readable messages on failure

79–81

ditto

vlorentz requested changes to this revision.Nov 4 2022, 9:27 AM
This revision now requires changes to proceed.Nov 4 2022, 9:27 AM
franckbret updated this revision to Diff 31764.Nov 4 2022, 5:21 PM
franckbret marked 6 inline comments as done.

minor improvments

swh-public-ci added a comment.Nov 4 2022, 5:26 PM

Build is green

Patch application report for D8748 (id=31764)

Rebasing onto 60707a45dd...

First, rewinding head to replay your work on top of it...
Applying: Nuget: Implement incremental listing
Changes applied before test
commit 13fe0fef3162683529c102d51a219a40dab0a138
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 19 16:11:23 2022 +0200

    Nuget: Implement incremental listing
    
    The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/848/ for more details.

Harbormaster completed remote builds in B32717: Diff 31764.Nov 4 2022, 5:26 PM
franckbret added inline comments.Nov 7 2022, 8:40 AM
swh/lister/nuget/lister.py
147–148

No it can't. From https://learn.microsoft.com/en-us/nuget/api/catalog-resource#catalog-index

Each catalog object has a property called the commitTimeStamp representing when the item was added to the catalog. Catalog items are added to a catalog page in batches called commits.

vlorentz accepted this revision.Nov 10 2022, 1:06 PM
This revision is now accepted and ready to land.Nov 10 2022, 1:06 PM
franckbret updated this revision to Diff 31841.Nov 14 2022, 9:33 AM

Rebase

This revision was landed with ongoing or failed builds.Nov 14 2022, 9:33 AM
Closed by commit rDLSea146ce297d5: Nuget: Implement incremental listing (authored by franckbret). · Explain Why
This revision was automatically updated to reflect the committed changes.
franckbret added a commit: rDLSea146ce297d5: Nuget: Implement incremental listing.
swh-public-ci added a comment.Nov 14 2022, 9:39 AM

Build is green

Patch application report for D8748 (id=31841)

Rebasing onto e1f3f87c73...

Current branch diff-target is up to date.
Changes applied before test
commit ea146ce297d55fb73f2975c546eebbf3cbaec295
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 19 16:11:23 2022 +0200

    Nuget: Implement incremental listing
    
    The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/858/ for more details.

Harbormaster completed remote builds in B32778: Diff 31841.Nov 14 2022, 9:39 AM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
lister/
nuget/
13 lines83%
Loading...
78 lines91%
Loading...
tests/
data/
https_api.nuget.org/
25 lines--
187 lines--
46 lines--
49 lines--
102 lines100%
Loading...

Diff 31566

View Options

swh/lister/nuget/__init__.py

Loading...
View Options

swh/lister/nuget/lister.py

Loading...
View Options

swh/lister/nuget/tests/data/https_api.nuget.org/v3-flatcontainer_moq.automock_3.5.0-ci0287_moq.automock.nuspec

Loading...
View Options

swh/lister/nuget/tests/data/https_api.nuget.org/v3_catalog0_data_2022.10.10.04.04.00_moq.automock.3.5.0-ci0287.json

Loading...
View Options

swh/lister/nuget/tests/data/https_api.nuget.org/v3_catalog0_index.json_visit1

Loading...
View Options

swh/lister/nuget/tests/data/https_api.nuget.org/v3_catalog0_page17100.json

Loading...
View Options

swh/lister/nuget/tests/test_lister.py

Loading...
About · Developer information · Legal