Page MenuHomeSoftware Heritage
Differential D7879

Adapt maven lister to list canonical gh urls if any
ClosedPublic
Actions

Authored by ardumont on May 20 2022, 4:38 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T4232: Listers: Canonicalize listed github origins
Commits
rDLS263db667d09c: Adapt maven lister to list canonical gh urls if any
Summary

That means detected github urls {https,git,http}://github.com/${user_repo}(.git) are
canonicalized to https://github.com/${user_repo} format.

This avoids duplication of origins.

Related to T4232
Depends on D7880

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.May 20 2022, 4:38 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 20 2022, 4:38 PM
ardumont mentioned this in T4232: Listers: Canonicalize listed github origins.May 20 2022, 4:38 PM
swh-public-ci added a comment.May 20 2022, 4:42 PM

Build has FAILED

Patch application report for D7879 (id=28432)

Rebasing onto 2ffe9c2aea...

Current branch diff-target is up to date.
Changes applied before test
commit f9bf6aefb1a4ef407d6f64bb4cbea778a9667edc
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri May 20 16:37:15 2022 +0200

    Adapt maven lister to list canonical gh urls if any
    
    Related to T4232

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/524/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/524/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 20 2022, 4:42 PM
Harbormaster failed remote builds in B29523: Diff 28432!
ardumont updated this revision to Diff 28433.May 20 2022, 4:52 PM

Add missing pytest configuration (mock github api call)

swh-public-ci added a comment.May 20 2022, 4:56 PM

Build has FAILED

Patch application report for D7879 (id=28433)

Rebasing onto 2ffe9c2aea...

Current branch diff-target is up to date.
Changes applied before test
commit c7d4faae7c81ef2fc1230738ff42cb56a771e0d2
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri May 20 16:37:15 2022 +0200

    Adapt maven lister to list canonical gh urls if any
    
    Related to T4232

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/525/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/525/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 20 2022, 4:56 PM
Harbormaster failed remote builds in B29524: Diff 28433!
ardumont updated this revision to Diff 28434.May 20 2022, 4:58 PM

Fix requests_mock override definition

swh-public-ci added a comment.May 20 2022, 5:02 PM

Build has FAILED

Patch application report for D7879 (id=28434)

Rebasing onto 2ffe9c2aea...

Current branch diff-target is up to date.
Changes applied before test
commit c2c4d7d440ef512f3424748a33ef8b001b6311b7
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri May 20 16:37:15 2022 +0200

    Adapt maven lister to list canonical gh urls if any
    
    Related to T4232

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/526/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/526/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 20 2022, 5:02 PM
Harbormaster failed remote builds in B29525: Diff 28434!
ardumont updated this revision to Diff 28438.May 20 2022, 5:48 PM

Deal with canonical urls

swh-public-ci added a comment.May 20 2022, 5:52 PM

Build has FAILED

Patch application report for D7879 (id=28438)

Rebasing onto 2ffe9c2aea...

Current branch diff-target is up to date.
Changes applied before test
commit 6fa40a1d4c32746f30d2180c67901b888ced9187
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri May 20 16:37:15 2022 +0200

    Adapt maven lister to list canonical gh urls if any
    
    Related to T4232

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/527/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/527/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 20 2022, 5:52 PM
Harbormaster failed remote builds in B29529: Diff 28438!
ardumont edited the summary of this revision. (Show Details)May 20 2022, 5:59 PM
ardumont added a parent revision: D7880: Deal with git protocol url to canonicalize to https.
ardumont edited the test plan for this revision. (Show Details)
ardumont requested review of this revision.May 20 2022, 6:03 PM
ardumont edited the test plan for this revision. (Show Details)
ardumont edited the test plan for this revision. (Show Details)
ardumont retitled this revision from wip: Adapt maven lister to list canonical gh urls if any to Adapt maven lister to list canonical gh urls if any.
ardumont updated this revision to Diff 28441.May 20 2022, 6:28 PM
ardumont edited the test plan for this revision. (Show Details)

Bump requirements, tests should pass now.

ardumont edited the test plan for this revision. (Show Details)May 20 2022, 6:29 PM
ardumont updated this revision to Diff 28442.May 20 2022, 6:29 PM

What i said but using the right repository this time ¯\_(ツ)_/¯

swh-public-ci added a comment.May 20 2022, 6:31 PM

Build is green

Patch application report for D7879 (id=28441)

Rebasing onto 817c9e099c...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-427-D7879.
Changes applied before test



See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/427/ for more details.
Harbormaster completed remote builds in B29532: Diff 28441.May 20 2022, 6:31 PM
swh-public-ci added a comment.May 20 2022, 6:33 PM
Build is green



Patch application report for D7879 (id=28442)



Rebasing onto 2ffe9c2aea...



Current branch diff-target is up to date.





Changes applied before test



commit 7f8c778fd7f315641fac8c4507a6fc9eb8d8de9d
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri May 20 16:37:15 2022 +0200

    Adapt maven lister to list canonical gh urls if any
    
    Related to T4232



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/528/ for more details.
Harbormaster completed remote builds in B29533: Diff 28442.May 20 2022, 6:33 PM
ardumont edited the summary of this revision. (Show Details)May 23 2022, 9:21 AM
ardumont updated this revision to Diff 28445.May 23 2022, 9:24 AM
    

  • Improve commit message
    • 

  • Reword docstrings
    • 

  • Add comments
    • 

swh-public-ci added a comment.May 23 2022, 9:29 AM
Build is green



Patch application report for D7879 (id=28445)



Rebasing onto 2ffe9c2aea...



Current branch diff-target is up to date.





Changes applied before test



commit 0a997670d4c01ba11f5c7cfb27a13bc1be1ded58
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri May 20 16:37:15 2022 +0200

    Adapt maven lister to list canonical gh urls if any
    
    That means detected github urls {https,git,http}://github.com/${user_repo}(.git) are
    canonicalized to https://github.com/${user_repo} format.
    
    This avoids duplication of origins.
    
    Related to T4232



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/529/ for more details.
Harbormaster completed remote builds in B29537: Diff 28445.May 23 2022, 9:29 AM
vlorentz added a subscriber: vlorentz.May 23 2022, 11:37 AM
vlorentz added inline comments.
swh/lister/maven/lister.py


58
could you move it to the toplevel? it doesn't need to be an attribute


303–322
It is simpler to return early than to re-check the same conditions every time, IMO
ardumont added inline comments.May 23 2022, 11:48 AM
swh/lister/maven/lister.py


58
sure


303–322
yeah, i was missing the else return None you put early on, thanks.
ardumont updated this revision to Diff 28447.May 23 2022, 11:54 AM
Adapt according to review
swh-public-ci added a comment.May 23 2022, 11:58 AM
Build has FAILED



Patch application report for D7879 (id=28447)



Rebasing onto 2ffe9c2aea...



Current branch diff-target is up to date.





Changes applied before test



commit 114074a0ead6e38cef7f6b0dba37cd7f65161ef3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri May 20 16:37:15 2022 +0200

    Adapt maven lister to list canonical gh urls if any
    
    That means detected github urls {https,git,http}://github.com/${user_repo}(.git) are
    canonicalized to https://github.com/${user_repo} format.
    
    This avoids duplication of origins.
    
    Related to T4232



Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/530/

See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/530/console
Harbormaster failed remote builds in B29538: Diff 28447!May 23 2022, 11:58 AM
vlorentz added inline comments.May 23 2022, 12:07 PM
swh/lister/maven/lister.py


314–319
another one
ardumont added inline comments.May 23 2022, 12:48 PM
swh/lister/maven/lister.py


314–319
No, here it must be after the get_canonical call because that could return None.

Hence why i put it there
vlorentz accepted this revision.May 23 2022, 1:37 PM
vlorentz added inline comments.
swh/lister/maven/lister.py


314–319
ack
This revision is now accepted and ready to land.May 23 2022, 1:37 PM
ardumont added a comment.May 23 2022, 2:23 PM
So now let's fix the unit tests.

I've managed to break the tests with the recent fixes.
ardumont updated this revision to Diff 28454.May 23 2022, 2:47 PM
Fix
swh-public-ci added a comment.May 23 2022, 2:51 PM
Build is green



Patch application report for D7879 (id=28454)



Rebasing onto 2ffe9c2aea...



Current branch diff-target is up to date.





Changes applied before test



commit 263db667d09c4090dbc9d7be1df5aae0721c612b
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Fri May 20 16:37:15 2022 +0200

    Adapt maven lister to list canonical gh urls if any
    
    That means detected github urls {https,git,http}://github.com/${user_repo}(.git) are
    canonicalized to https://github.com/${user_repo} format.
    
    This avoids duplication of origins.
    
    Related to T4232



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/531/ for more details.
Harbormaster completed remote builds in B29546: Diff 28454.May 23 2022, 2:51 PM
Closed by commit rDLS263db667d09c: Adapt maven lister to list canonical gh urls if any (authored by ardumont).  ·  Explain WhyMay 23 2022, 3:39 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDLS263db667d09c: Adapt maven lister to list canonical gh urls if any.
ardumont mentioned this in T3874: staging: Analyze result of the maven listing and ingestion.Jun 1 2022, 3:01 PM
Revision Contents
Changeset List
PathSize
2 lines
swh/
lister/
maven/
87 lines
tests/
39 lines
Diff 28455
View Options
requirements-swh.txt
Loading...
View Options
swh/lister/maven/lister.py
Loading...
View Options
swh/lister/maven/tests/test_lister.py
Loading...
About · Developer information · Legal