Page MenuHomeSoftware Heritage
Differential D7946

github/utils: Deal with exotic urls to canonicalize
ClosedPublic
Actions

Authored by ardumont on Jun 2 2022, 3:14 PM.

Details

Reviewers
anlambert
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T3874: staging: Analyze result of the maven listing and ingestion
Commits
rDCOREe1a1d84eb4ea: github/utils: Deal with exotic urls to canonicalize
Summary

The sample of those exotic urls got extracted out the staging scheduler [1].

And the actual run of that code makes the github api (without mocks) returns the correct
canonical urls [2]

Related to T3874

[1] P1371

[2]

$ ipython
...
In [1]: from swh.core.github.utils import get_canonical_github_origin_url

In [2]: get_canonical_github_origin_url('git@github.com/huaweicloud/huaweicloud-sdk-java-v3.git')
No tokens set in configuration, using anonymous mode
Out[2]: 'https://github.com/huaweicloud/huaweicloud-sdk-java-v3'

In [3]: get_canonical_github_origin_url('git//github.com/powertac/powertac-server.git')
No tokens set in configuration, using anonymous mode
Out[3]: 'https://github.com/powertac/powertac-server'

In [4]: get_canonical_github_origin_url('https://${env.GITHUB_USER}:${env.GITHUB_TOKEN}@github.com/molgenis/vibe.git')
No tokens set in configuration, using anonymous mode
Out[4]: 'https://github.com/molgenis/vibe'

In [5]: get_canonical_github_origin_url('ssh://git@github.com/softwaremagico/ThinkMachine.git')
No tokens set in configuration, using anonymous mode
Out[5]: 'https://github.com/softwaremagico/ThinkMachine'

In [6]: get_canonical_github_origin_url('ssh://github.com:alibaba/SmartEngine.git')
No tokens set in configuration, using anonymous mode
Out[6]: 'https://github.com/alibaba/SmartEngine'

In [7]: get_canonical_github_origin_url('//github.com:networknt/light-tram-kafka.git')
No tokens set in configuration, using anonymous mode
Out[7]: 'https://github.com/networknt/light-tram-kafka'

In [8]: get_canonical_github_origin_url('[fetch=]git@github.com:turnonline/ecosystem-admin-widgets.git')
No tokens set in configuration, using anonymous mode
Out[8]: 'https://github.com/turnonline/ecosystem-admin-widgets'

In [9]: get_canonical_github_origin_url('git@github.com:ttulka/spring-boot-configuration-properties-store.git')
No tokens set in configuration, using anonymous mode
Out[9]: 'https://github.com/ttulka/spring-boot-configuration-properties-store'

Diff Detail

Event Timeline

ardumont created this revision.Jun 2 2022, 3:14 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptJun 2 2022, 3:14 PM
swh-public-ci added a comment.Jun 2 2022, 3:17 PM

Build is green

Patch application report for D7946 (id=28615)

Rebasing onto 4fc5f601b6...

Current branch diff-target is up to date.
Changes applied before test
commit 4a32a4610d174545eff53c6753ee8fd55060dca5
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Jun 2 15:09:54 2022 +0200

    github/utils: Deal with exotic urls to canonicalize
    
    The sample of those exotic urls got extracted out the staging scheduler [1].
    
    And the actual run of that code makes the github api (without mocks) returns the correct
    canonical urls [2]
    
    Related to T3874
    
    [1] P1371
    
    [2]
$ ipython
...
In [1]: from swh.core.github.utils import get_canonical_github_origin_url

In [2]: get_canonical_github_origin_url('git@github.com/huaweicloud/huaweicloud-sdk-java-v3.git')
No tokens set in configuration, using anonymous mode
Out[2]: 'https://github.com/huaweicloud/huaweicloud-sdk-java-v3'

In [3]: get_canonical_github_origin_url('git//github.com/powertac/powertac-server.git')
No tokens set in configuration, using anonymous mode
Out[3]: 'https://github.com/powertac/powertac-server'

In [4]: get_canonical_github_origin_url('https://${env.GITHUB_USER}:${env.GITHUB_TOKEN}@github.com/molgenis/vibe.git')
No tokens set in configuration, using anonymous mode
Out[4]: 'https://github.com/molgenis/vibe'

In [5]: get_canonical_github_origin_url('ssh://git@github.com/softwaremagico/ThinkMachine.git')
No tokens set in configuration, using anonymous mode
Out[5]: 'https://github.com/softwaremagico/ThinkMachine'

In [6]: get_canonical_github_origin_url('ssh://github.com:alibaba/SmartEngine.git')
No tokens set in configuration, using anonymous mode
Out[6]: 'https://github.com/alibaba/SmartEngine'

In [7]: get_canonical_github_origin_url('//github.com:networknt/light-tram-kafka.git')
No tokens set in configuration, using anonymous mode
Out[7]: 'https://github.com/networknt/light-tram-kafka'
```
See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/439/ for more details.
Harbormaster completed remote builds in B29700: Diff 28615.Jun 2 2022, 3:17 PM
ardumont requested review of this revision.Jun 2 2022, 3:17 PM
ardumont updated this revision to Diff 28616.Jun 2 2022, 3:23 PM

Update tests

ardumont edited the summary of this revision. (Show Details)Jun 2 2022, 3:24 PM
ardumont added inline comments.
swh/core/github/utils.py
22

lies!

22

coverage report lies!

swh-public-ci added a comment.Jun 2 2022, 3:26 PM

Build is green

Patch application report for D7946 (id=28616)

Rebasing onto 4fc5f601b6...

Current branch diff-target is up to date.
Changes applied before test
commit 7923da2b8adb36f4885acf2ba6e953d626ef4513
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Jun 2 15:09:54 2022 +0200

    github/utils: Deal with exotic urls to canonicalize
    
    The sample of those exotic urls got extracted out the staging scheduler [1].
    
    And the actual run of that code makes the github api (without mocks) returns the correct
    canonical urls [2]
    
    Related to T3874
    
    [1] P1371
    
    [2]
$ ipython
...
In [1]: from swh.core.github.utils import get_canonical_github_origin_url

In [2]: get_canonical_github_origin_url('git@github.com/huaweicloud/huaweicloud-sdk-java-v3.git')
No tokens set in configuration, using anonymous mode
Out[2]: 'https://github.com/huaweicloud/huaweicloud-sdk-java-v3'

In [3]: get_canonical_github_origin_url('git//github.com/powertac/powertac-server.git')
No tokens set in configuration, using anonymous mode
Out[3]: 'https://github.com/powertac/powertac-server'

In [4]: get_canonical_github_origin_url('https://${env.GITHUB_USER}:${env.GITHUB_TOKEN}@github.com/molgenis/vibe.git')
No tokens set in configuration, using anonymous mode
Out[4]: 'https://github.com/molgenis/vibe'

In [5]: get_canonical_github_origin_url('ssh://git@github.com/softwaremagico/ThinkMachine.git')
No tokens set in configuration, using anonymous mode
Out[5]: 'https://github.com/softwaremagico/ThinkMachine'

In [6]: get_canonical_github_origin_url('ssh://github.com:alibaba/SmartEngine.git')
No tokens set in configuration, using anonymous mode
Out[6]: 'https://github.com/alibaba/SmartEngine'

In [7]: get_canonical_github_origin_url('//github.com:networknt/light-tram-kafka.git')
No tokens set in configuration, using anonymous mode
Out[7]: 'https://github.com/networknt/light-tram-kafka'

In [8]: get_canonical_github_origin_url('[fetch=]git@github.com:turnonline/ecosystem-admin-widgets.git')
No tokens set in configuration, using anonymous mode
Out[8]: 'https://github.com/turnonline/ecosystem-admin-widgets'

In [9]: get_canonical_github_origin_url('git@github.com:ttulka/spring-boot-configuration-properties-store.git')
No tokens set in configuration, using anonymous mode
Out[9]: 'https://github.com/ttulka/spring-boot-configuration-properties-store'

```
See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/440/ for more details.
Harbormaster completed remote builds in B29701: Diff 28616.Jun 2 2022, 3:26 PM
anlambert added a subscriber: anlambert.Jun 2 2022, 3:37 PM
anlambert added inline comments.
swh/core/github/utils.py
22

Just by curiosity, did you try to use urlparse from urllib.parse Python module instead of a regexp ?

>>> s = "//github.com/toto/titi"
>>> from urllib.parse import urlparse
>>> urlparse(s)
ParseResult(scheme='', netloc='github.com', path='/toto/titi', params='', query='', fragment='')

The user repo should be in the path field of ParseResult object, looks simpler to me.

vlorentz added a subscriber: vlorentz.Jun 2 2022, 3:39 PM
vlorentz added inline comments.
swh/core/github/tests/test_github_utils.py
20–21

it seems weird to have a function just for a docstring, but whatever

37

at this point, s/protocol/prefix/ here and in the function signature.

45

that's oddly specific, where does this example come from?

ardumont added inline comments.Jun 2 2022, 3:43 PM
swh/core/github/tests/test_github_utils.py
20–21

i did not get it.

37

yeah, better.

45

The paste mentioned in the diff description [1] (coming out of the full maven listing).

[1] P1371

swh/core/github/utils.py
22

not tested at all, thx for the idea.

ardumont added inline comments.Jun 2 2022, 3:53 PM
swh/core/github/tests/test_github_utils.py
45

fwiw, maven listing still ongoing

swh/core/github/utils.py
22

unfortunately, It won't work for other ones:

In [1]: url='git@github.com/user/repo.git'

In [2]: from urllib.parse import urlparse

In [3]: urlparse(url)
Out[3]: ParseResult(scheme='', netloc='', path='git@github.com/user/repo.git', params='', query='', fragment='')
ardumont marked 2 inline comments as done.Jun 2 2022, 3:55 PM
vlorentz added inline comments.Jun 2 2022, 3:57 PM
swh/core/github/tests/test_github_utils.py
20–21

s/docstring/f-string/

ardumont updated this revision to Diff 28623.Jun 2 2022, 3:58 PM

Drop function for f-string, simpler indeed.

anlambert accepted this revision.Jun 2 2022, 3:58 PM

Looks good to me.

swh/core/github/utils.py
22

ack, too much garbage in maven listing ;-)

This revision is now accepted and ready to land.Jun 2 2022, 3:58 PM
ardumont added inline comments.Jun 2 2022, 3:58 PM
swh/core/github/utils.py
22

lolsob! yeah!

ardumont marked 3 inline comments as done.Jun 2 2022, 3:59 PM
vlorentz accepted this revision.Jun 2 2022, 4:00 PM
swh-public-ci added a comment.Jun 2 2022, 4:00 PM

Build is green

Patch application report for D7946 (id=28623)

Rebasing onto 4fc5f601b6...

Current branch diff-target is up to date.
Changes applied before test
commit e1a1d84eb4eaa73500636c1132196ab025542ea8
Author: Antoine R. Dumont (@ardumont) <antoine.romain.dumont@gmail.com>
Date:   Thu Jun 2 15:09:54 2022 +0200

    github/utils: Deal with exotic urls to canonicalize
    
    The sample of those exotic urls got extracted out the staging scheduler [1].
    
    And the actual run of that code makes the github api (without mocks) returns the correct
    canonical urls [2]
    
    Related to T3874
    
    [1] P1371
    
    [2]
$ ipython
...
In [1]: from swh.core.github.utils import get_canonical_github_origin_url

In [2]: get_canonical_github_origin_url('git@github.com/huaweicloud/huaweicloud-sdk-java-v3.git')
No tokens set in configuration, using anonymous mode
Out[2]: 'https://github.com/huaweicloud/huaweicloud-sdk-java-v3'

In [3]: get_canonical_github_origin_url('git//github.com/powertac/powertac-server.git')
No tokens set in configuration, using anonymous mode
Out[3]: 'https://github.com/powertac/powertac-server'

In [4]: get_canonical_github_origin_url('https://${env.GITHUB_USER}:${env.GITHUB_TOKEN}@github.com/molgenis/vibe.git')
No tokens set in configuration, using anonymous mode
Out[4]: 'https://github.com/molgenis/vibe'

In [5]: get_canonical_github_origin_url('ssh://git@github.com/softwaremagico/ThinkMachine.git')
No tokens set in configuration, using anonymous mode
Out[5]: 'https://github.com/softwaremagico/ThinkMachine'

In [6]: get_canonical_github_origin_url('ssh://github.com:alibaba/SmartEngine.git')
No tokens set in configuration, using anonymous mode
Out[6]: 'https://github.com/alibaba/SmartEngine'

In [7]: get_canonical_github_origin_url('//github.com:networknt/light-tram-kafka.git')
No tokens set in configuration, using anonymous mode
Out[7]: 'https://github.com/networknt/light-tram-kafka'

In [8]: get_canonical_github_origin_url('[fetch=]git@github.com:turnonline/ecosystem-admin-widgets.git')
No tokens set in configuration, using anonymous mode
Out[8]: 'https://github.com/turnonline/ecosystem-admin-widgets'

In [9]: get_canonical_github_origin_url('git@github.com:ttulka/spring-boot-configuration-properties-store.git')
No tokens set in configuration, using anonymous mode
Out[9]: 'https://github.com/ttulka/spring-boot-configuration-properties-store'

```
See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/441/ for more details.
Harbormaster completed remote builds in B29706: Diff 28623.Jun 2 2022, 4:00 PM
Closed by commit rDCOREe1a1d84eb4ea: github/utils: Deal with exotic urls to canonicalize (authored by ardumont). · Explain WhyJun 2 2022, 4:01 PM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDCOREe1a1d84eb4ea: github/utils: Deal with exotic urls to canonicalize.
ardumont mentioned this in T3874: staging: Analyze result of the maven listing and ingestion.Jun 3 2022, 9:35 AM
ardumont mentioned this in T4232: Listers: Canonicalize listed github origins.Jun 3 2022, 3:17 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
core/
github/
tests/
94 lines100%
Loading...
4 lines94%
Loading...

Diff 28623

View Options

swh/core/github/tests/test_github_utils.py

Loading...
View Options

swh/core/github/utils.py

Loading...
About · Developer information · Legal