Page MenuHomeSoftware Heritage
Differential D8673

packagist: Canonicalize github origins
ClosedPublic
Actions

Authored by vlorentz on Oct 13 2022, 1:06 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Commits
rDLSa681f2f40526: packagist: Canonicalize github origins
Summary

In particular, there seems to be a negligeable number of origins
using SSH instead of HTTPS, which the git loader cannot deal with.

https://sentry.softwareheritage.org/share/issue/709325715f6d44cdbb7cc27a244a472a/

Test Plan

Build fails because it depends on D8674

Diff Detail

Repository
rDLS Listers
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

vlorentz created this revision.Oct 13 2022, 1:06 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 13 2022, 1:06 PM
swh-public-ci added a comment.Oct 13 2022, 1:12 PM

Build is green

Patch application report for D8673 (id=31327)

Rebasing onto f5c5599f2e...

Current branch diff-target is up to date.
Changes applied before test
commit e52c0183076898fb0d8c6b3e174b365bd60196f2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/792/ for more details.

Harbormaster completed remote builds in B32277: Diff 31327.Oct 13 2022, 1:12 PM
vlorentz requested review of this revision.Oct 13 2022, 1:12 PM
anlambert requested changes to this revision.Oct 13 2022, 1:21 PM
anlambert added a subscriber: anlambert.
anlambert added inline comments.
swh/lister/packagist/lister.py
66

I think you forgot the call to self.github_sesssion.get_canonical_url.

This revision now requires changes to proceed.Oct 13 2022, 1:21 PM
douardda added a subscriber: douardda.Oct 13 2022, 1:50 PM
douardda added inline comments.
swh/lister/packagist/lister.py
66

how can the test be green then?

anlambert added inline comments.Oct 13 2022, 1:51 PM
swh/lister/packagist/lister.py
66

because the origin URL is already canonicalized in tests data.

vlorentz added a comment.Oct 13 2022, 2:22 PM

that'll teach me to TDD

but also wtf, I don't see why the URL is already canon when I query the Packagist API, but the lister somehow used a SSH URL

vlorentz updated this revision to Diff 31334.Oct 13 2022, 4:16 PM
  • actually implement it
  • use a non-canon URL in test
  • add a gitlab-based project in test data (instead of only github)
vlorentz updated this revision to Diff 31335.Oct 13 2022, 4:18 PM

Test the case of deleted github repos

swh/lister/packagist/tests/data/https_api.github.com/repos_gitlky_wx_article
1

the original project was deleted, so I wrote this manually

swh-public-ci added a comment.Oct 13 2022, 4:21 PM

Build has FAILED

Patch application report for D8673 (id=31334)

Rebasing onto f5c5599f2e...

Current branch diff-target is up to date.
Changes applied before test
commit c7454c6a205e3f048dae006b4137d2031296fbcd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/793/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/793/console

Harbormaster failed remote builds in B32282: Diff 31334!Oct 13 2022, 4:22 PM
swh-public-ci added a comment.Oct 13 2022, 4:24 PM

Build has FAILED

Patch application report for D8673 (id=31335)

Rebasing onto f5c5599f2e...

Current branch diff-target is up to date.
Changes applied before test
commit a3ba391ab0bd645cb0d61a48234d03847465d6a5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/794/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/794/console

Harbormaster failed remote builds in B32283: Diff 31335!Oct 13 2022, 4:24 PM
vlorentz edited the test plan for this revision. (Show Details)Oct 13 2022, 4:28 PM
vlorentz added a parent revision: D8674: Fix _sanitize_github_url removing suffixes too greedily.
anlambert accepted this revision.Oct 13 2022, 4:34 PM
This revision is now accepted and ready to land.Oct 13 2022, 4:34 PM
vlorentz updated this revision to Diff 31344.Oct 13 2022, 4:49 PM

bump dependency on swh-core

swh-public-ci added a comment.Oct 13 2022, 4:49 PM

Build has FAILED

Patch application report for D8673 (id=31344)

Rebasing onto 82b936a277...

First, rewinding head to replay your work on top of it...
Applying: packagist: Canonicalize github origins
Changes applied before test
commit a325693d9103d2d04f8f1cea55f210fe814e5ff6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/795/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/795/console

Harbormaster failed remote builds in B32292: Diff 31344!Oct 13 2022, 4:49 PM
swh-public-ci added a comment.Oct 13 2022, 4:56 PM

Build is green

Patch application report for D8673 (id=31344)

Rebasing onto 82b936a277...

First, rewinding head to replay your work on top of it...
Applying: packagist: Canonicalize github origins
Changes applied before test
commit 17adddd988fd2bdbce7e8070ec2640fbef441779
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/796/ for more details.

Harbormaster completed remote builds in B32292: Diff 31344.Oct 13 2022, 4:56 PM
vlorentz updated this revision to Diff 31345.Oct 13 2022, 5:15 PM

rebase

This revision was landed with ongoing or failed builds.Oct 13 2022, 5:15 PM
Closed by commit rDLSa681f2f40526: packagist: Canonicalize github origins (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDLSa681f2f40526: packagist: Canonicalize github origins.
swh-public-ci added a comment.Oct 13 2022, 5:21 PM

Build is green

Patch application report for D8673 (id=31345)

Rebasing onto a681f2f405...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-797-D8673.
Changes applied before test



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/797/ for more details.
Harbormaster completed remote builds in B32293: Diff 31345.Oct 13 2022, 5:21 PM
Revision Contents
Changeset List
PathSize
2 lines
swh/
lister/
packagist/
12 lines
tests/
data/
https_api.github.com/
1 line
130 lines
104 lines
309 lines
41 lines
46 lines
Diff 31346
View Options
requirements-swh.txt
Loading...
View Options
swh/lister/packagist/lister.py
Loading...
View Options
swh/lister/packagist/tests/data/https_api.github.com/repos_gitlky_wx_article
Loading...
View Options
swh/lister/packagist/tests/data/https_api.github.com/repos_spryker-eco_computop-api
Loading...
View Options
swh/lister/packagist/tests/data/https_api.github.com/repos_ycms_module-main
Loading...
View Options
swh/lister/packagist/tests/data/idevlab_essential.json
Loading...
View Options
swh/lister/packagist/tests/data/ycms_module-main.json
Loading...
View Options
swh/lister/packagist/tests/test_lister.py
Loading...
About · Developer information · Legal