Page MenuHomeSoftware Heritage
Differential D8772

metadata_dictionary: Systematically check input URLs before adding to graph
ClosedPublic
Actions

Authored by vlorentz on Oct 25 2022, 4:03 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Maniphest Tasks
T4656: AttributeError: 'NoneType' object has no attribute 'endswith'
Commits
rDCIDXa66d5b240ab7: metadata_dictionary: Systematically check input URLs before adding to graph
Summary

This is hopefully the definitive workaround for the PyLD issue.

Closes T4656

Diff Detail

Repository
rDCIDX Metadata indexer
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 32574
Build 51026: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 51025: arc lint + arc unit

Unit TestsFailed
View All

TimeTest
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.maven::swh.indexer.metadata_dictionary.maven.MavenMapping.translate_licenses
106 { 107 "licenses": { 108 "license": {
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.codemeta::swh.indexer.codemeta.make_absolute_uri
9 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.gitea::swh.indexer.metadata_dictionary.gitea.GiteaMapping.translate_forks_count
3 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.gitea::swh.indexer.metadata_dictionary.gitea.GiteaMapping.translate_stars_count
4 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.indexer.metadata_dictionary.gitea::swh.indexer.metadata_dictionary.gitea.GiteaMapping.translate_watchers_count
View Full Test Results (1 Failed · 366 Passed · 11 Skipped)

Event Timeline

vlorentz created this revision.Oct 25 2022, 4:03 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptOct 25 2022, 4:03 PM
swh-public-ci added a comment.Oct 25 2022, 4:07 PM

Build has FAILED

Patch application report for D8772 (id=31621)

Rebasing onto a51cbf3965...

Current branch diff-target is up to date.
Changes applied before test
commit 4148da3ef7f6b7a5b98151aa6179502842f215ba
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Oct 25 16:02:16 2022 +0200

    metadata_dictionary: Systematically check input URLs before adding to graph
    
    This is hopefully the definitive workaround for the PyLD issue.

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/517/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/517/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 25 2022, 4:07 PM
Harbormaster failed remote builds in B32574: Diff 31621!
vlorentz updated this revision to Diff 31622.Oct 25 2022, 4:22 PM

flake8 lied

swh-public-ci added a comment.Oct 25 2022, 4:26 PM

Build is green

Patch application report for D8772 (id=31622)

Rebasing onto a51cbf3965...

Current branch diff-target is up to date.
Changes applied before test
commit a66d5b240ab77e6d8d1b9accf43d571489a3f7f0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Oct 25 16:02:16 2022 +0200

    metadata_dictionary: Systematically check input URLs before adding to graph
    
    This is hopefully the definitive workaround for the PyLD issue.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/518/ for more details.

Harbormaster completed remote builds in B32575: Diff 31622.Oct 25 2022, 4:26 PM
vlorentz requested review of this revision.Oct 25 2022, 4:27 PM
vlorentz added a child revision: D8778: codemeta: Fix incorrect output namespace for dates and URLs.Oct 26 2022, 2:20 PM
anlambert accepted this revision.Oct 26 2022, 3:14 PM
anlambert added a subscriber: anlambert.

LGTM, added some nitpicks about typing as inline comments.

swh/indexer/metadata_dictionary/utils.py
79

Typing could be more precise here.

url: Optional[str]
107–110

Could be merged into a single if block.

if url is None or " " in url or not urllib.parse.urlparse(url).netloc:
        return
This revision is now accepted and ready to land.Oct 26 2022, 3:14 PM
vlorentz added inline comments.Oct 26 2022, 4:56 PM
swh/indexer/metadata_dictionary/utils.py
79

not really, url comes from arbitrary JSON, YAML, or XML files.

Closed by commit rDCIDXa66d5b240ab7: metadata_dictionary: Systematically check input URLs before adding to graph (authored by vlorentz). · Explain WhyOct 27 2022, 2:23 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDCIDXa66d5b240ab7: metadata_dictionary: Systematically check input URLs before adding to graph.

Revision Contents
Changeset List

PathSize
swh/
indexer/
metadata_dictionary/
25 lines
7 lines
13 lines
10 lines
16 lines
4 lines
42 lines
tests/
metadata_dictionary/
11 lines

Diff 31621

View Options

swh/indexer/metadata_dictionary/base.py

Loading...
View Options

swh/indexer/metadata_dictionary/cff.py

Loading...
View Options

swh/indexer/metadata_dictionary/github.py

Loading...
View Options

swh/indexer/metadata_dictionary/maven.py

Loading...
View Options

swh/indexer/metadata_dictionary/npm.py

Loading...
View Options

swh/indexer/metadata_dictionary/nuget.py

Loading...
View Options

swh/indexer/metadata_dictionary/utils.py

Loading...
View Options

swh/indexer/tests/metadata_dictionary/test_npm.py

Loading...
About · Developer information · Legal