Page MenuHomeSoftware Heritage
Differential D8673

packagist: Canonicalize github origins
ClosedPublic
Actions

Authored by vlorentz on Oct 13 2022, 1:06 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Commits
rDLSa681f2f40526: packagist: Canonicalize github origins
Summary

In particular, there seems to be a negligeable number of origins
using SSH instead of HTTPS, which the git loader cannot deal with.

https://sentry.softwareheritage.org/share/issue/709325715f6d44cdbb7cc27a244a472a/

Test Plan

Build fails because it depends on D8674

Diff Detail

Repository
rDLS Listers
Branch
packagist
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32283
Build 50567: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 50566: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
374 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.packagist.tests.test_lister::test_packagist_lister
swh_scheduler = <swh.scheduler.backend.SchedulerBackend object at 0x7f56c6d25828> requests_mock = <requests_mock.mocker.Mocker object at 0x7f56c550f4a8> datadir = '/var/lib/jenkins/workspace/DLS/tests-on-diff@2/.tox/py3/lib/python3.7/site-packages/swh/lister/packagist/tests/data'
2 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.lister::swh.lister.arch.lister.size_to_bytes
2,270 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_lister::test_arch_lister
1,006 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_tasks::test_arch_lister
4,170 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.lister.arch.tests.test_tasks::test_arch_ping
View Full Test Results (1 Failed · 335 Passed)

Event Timeline

vlorentz created this revision.Oct 13 2022, 1:06 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 13 2022, 1:06 PM
swh-public-ci added a comment.Oct 13 2022, 1:12 PM

Build is green

Patch application report for D8673 (id=31327)

Rebasing onto f5c5599f2e...

Current branch diff-target is up to date.
Changes applied before test
commit e52c0183076898fb0d8c6b3e174b365bd60196f2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/792/ for more details.

Harbormaster completed remote builds in B32277: Diff 31327.Oct 13 2022, 1:12 PM
vlorentz requested review of this revision.Oct 13 2022, 1:12 PM
anlambert requested changes to this revision.Oct 13 2022, 1:21 PM
anlambert added a subscriber: anlambert.
anlambert added inline comments.
swh/lister/packagist/lister.py
66

I think you forgot the call to self.github_sesssion.get_canonical_url.

This revision now requires changes to proceed.Oct 13 2022, 1:21 PM
douardda added a subscriber: douardda.Oct 13 2022, 1:50 PM
douardda added inline comments.
swh/lister/packagist/lister.py
66

how can the test be green then?

anlambert added inline comments.Oct 13 2022, 1:51 PM
swh/lister/packagist/lister.py
66

because the origin URL is already canonicalized in tests data.

vlorentz added a comment.Oct 13 2022, 2:22 PM

that'll teach me to TDD

but also wtf, I don't see why the URL is already canon when I query the Packagist API, but the lister somehow used a SSH URL

vlorentz updated this revision to Diff 31334.Oct 13 2022, 4:16 PM
  • actually implement it
  • use a non-canon URL in test
  • add a gitlab-based project in test data (instead of only github)
vlorentz updated this revision to Diff 31335.Oct 13 2022, 4:18 PM

Test the case of deleted github repos

swh/lister/packagist/tests/data/https_api.github.com/repos_gitlky_wx_article
1

the original project was deleted, so I wrote this manually

swh-public-ci added a comment.Oct 13 2022, 4:21 PM

Build has FAILED

Patch application report for D8673 (id=31334)

Rebasing onto f5c5599f2e...

Current branch diff-target is up to date.
Changes applied before test
commit c7454c6a205e3f048dae006b4137d2031296fbcd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/793/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/793/console

Harbormaster failed remote builds in B32282: Diff 31334!Oct 13 2022, 4:22 PM
swh-public-ci added a comment.Oct 13 2022, 4:24 PM

Build has FAILED

Patch application report for D8673 (id=31335)

Rebasing onto f5c5599f2e...

Current branch diff-target is up to date.
Changes applied before test
commit a3ba391ab0bd645cb0d61a48234d03847465d6a5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/794/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/794/console

Harbormaster failed remote builds in B32283: Diff 31335!Oct 13 2022, 4:24 PM
vlorentz edited the test plan for this revision. (Show Details)Oct 13 2022, 4:28 PM
vlorentz added a parent revision: D8674: Fix _sanitize_github_url removing suffixes too greedily.
anlambert accepted this revision.Oct 13 2022, 4:34 PM
This revision is now accepted and ready to land.Oct 13 2022, 4:34 PM
vlorentz updated this revision to Diff 31344.Oct 13 2022, 4:49 PM

bump dependency on swh-core

swh-public-ci added a comment.Oct 13 2022, 4:49 PM

Build has FAILED

Patch application report for D8673 (id=31344)

Rebasing onto 82b936a277...

First, rewinding head to replay your work on top of it...
Applying: packagist: Canonicalize github origins
Changes applied before test
commit a325693d9103d2d04f8f1cea55f210fe814e5ff6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

Link to build: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/795/
See console output for more information: https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/795/console

Harbormaster failed remote builds in B32292: Diff 31344!Oct 13 2022, 4:49 PM
swh-public-ci added a comment.Oct 13 2022, 4:56 PM

Build is green

Patch application report for D8673 (id=31344)

Rebasing onto 82b936a277...

First, rewinding head to replay your work on top of it...
Applying: packagist: Canonicalize github origins
Changes applied before test
commit 17adddd988fd2bdbce7e8070ec2640fbef441779
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Oct 13 13:05:34 2022 +0200

    packagist: Canonicalize github origins
    
    In particular, there seems to be a negligeable number of origins
    using SSH instead of HTTPS, which the git loader cannot deal with.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/796/ for more details.

Harbormaster completed remote builds in B32292: Diff 31344.Oct 13 2022, 4:56 PM
vlorentz updated this revision to Diff 31345.Oct 13 2022, 5:15 PM

rebase

This revision was landed with ongoing or failed builds.Oct 13 2022, 5:15 PM
Closed by commit rDLSa681f2f40526: packagist: Canonicalize github origins (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDLSa681f2f40526: packagist: Canonicalize github origins.
swh-public-ci added a comment.Oct 13 2022, 5:21 PM

Build is green

Patch application report for D8673 (id=31345)

Rebasing onto a681f2f405...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-797-D8673.
Changes applied before test



See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/797/ for more details.
Harbormaster completed remote builds in B32293: Diff 31345.Oct 13 2022, 5:21 PM
Revision Contents
Changeset List
PathSize
swh/
lister/
packagist/
12 lines
tests/
data/
https_api.github.com/
1 line
130 lines
104 lines
309 lines
41 lines
46 lines
Diff 31335
View Options
swh/lister/packagist/lister.py
Loading...
View Options
swh/lister/packagist/tests/data/https_api.github.com/repos_gitlky_wx_article
Loading...
View Options
swh/lister/packagist/tests/data/https_api.github.com/repos_spryker-eco_computop-api
Loading...
View Options
swh/lister/packagist/tests/data/https_api.github.com/repos_ycms_module-main
Loading...
View Options
swh/lister/packagist/tests/data/idevlab_essential.json
Loading...
View Options
swh/lister/packagist/tests/data/ycms_module-main.json
Loading...
View Options
swh/lister/packagist/tests/test_lister.py
Loading...
About · Developer information · Legal