Page MenuHomeSoftware Heritage

Nuget: Implement incremental listing
ClosedPublic

Authored by franckbret on Oct 19 2022, 4:15 PM.

Details

Summary

The lister is incremental and based on the value of `commitTimeStamp` retrieved on index http api endpoint.

Related T1718

Diff Detail

Repository
rDLS Listers
Branch
nuget-incremental
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32717
Build 51261: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 51260: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D8748 (id=31528)

Rebasing onto 0baaf68cff...

First, rewinding head to replay your work on top of it...
Applying: Nuget: Implement incremental listing
Changes applied before test
commit f67d9e2ffcee59725ee3e8a66bc6673fc7dc560e
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 19 16:11:23 2022 +0200

    Nuget: Implement incremental listing
    
    The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/804/ for more details.

it's unclear to me whether this actually implements the iteration protocol described in the API doc (https://learn.microsoft.com/en-us/nuget/api/catalog-resource#iterating-over-catalog-items) or not.

it's unclear to me whether this actually implements the iteration protocol described in the API doc (https://learn.microsoft.com/en-us/nuget/api/catalog-resource#iterating-over-catalog-items) or not.

To be clear, I think it should and it should be mentioned somewhere in a comment/docstring that this is what the code does, with a link to this precise part of the API documentation.

Improve documentation section related to incremental listing

Build is green

Patch application report for D8748 (id=31566)

Rebasing onto 8a82bbf95f...

First, rewinding head to replay your work on top of it...
Applying: Nuget: Implement incremental listing
Changes applied before test
commit 7b67c70c37a33ec7c36b1395b2049ff588e0a69f
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 19 16:11:23 2022 +0200

    Nuget: Implement incremental listing
    
    The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/810/ for more details.

Nice to have iteration documented officially by the API for once :)

Looks good overall, but some minor comments below, as usual

swh/lister/nuget/__init__.py
71–76
swh/lister/nuget/lister.py
78

unless it can be absent (in which case you need to do something with it instead of passing it to iso8601.parse_date)

88–90

(not A) or (A and B) is equivalent to (not A) or B

146–147

it is possible for commitTimeStamp to be missing? if yes, please add a test for it

swh/lister/nuget/tests/test_lister.py
49

remove this assertion, it's redundant with the next one, and gives less readable messages on failure

78–80

ditto

vlorentz requested changes to this revision.Nov 4 2022, 9:27 AM
This revision now requires changes to proceed.Nov 4 2022, 9:27 AM
franckbret marked 6 inline comments as done.

minor improvments

Build is green

Patch application report for D8748 (id=31764)

Rebasing onto 60707a45dd...

First, rewinding head to replay your work on top of it...
Applying: Nuget: Implement incremental listing
Changes applied before test
commit 13fe0fef3162683529c102d51a219a40dab0a138
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 19 16:11:23 2022 +0200

    Nuget: Implement incremental listing
    
    The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/848/ for more details.

swh/lister/nuget/lister.py
146–147

No it can't. From https://learn.microsoft.com/en-us/nuget/api/catalog-resource#catalog-index

Each catalog object has a property called the commitTimeStamp representing when the item was added to the catalog. Catalog items are added to a catalog page in batches called commits.

This revision is now accepted and ready to land.Nov 10 2022, 1:06 PM
This revision was landed with ongoing or failed builds.Nov 14 2022, 9:33 AM
This revision was automatically updated to reflect the committed changes.

Build is green

Patch application report for D8748 (id=31841)

Rebasing onto e1f3f87c73...

Current branch diff-target is up to date.
Changes applied before test
commit ea146ce297d55fb73f2975c546eebbf3cbaec295
Author: Franck Bret <franck.bret@octobus.net>
Date:   Wed Oct 19 16:11:23 2022 +0200

    Nuget: Implement incremental listing
    
    The lister is incremental and based on the value of ``commitTimeStamp`` retrieved on index http api endpoint.
    
    Related T1718

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/858/ for more details.