metadata_dictionary: Systematically check input URLs before adding to graph
ClosedPublic
Actions

Authored by vlorentz on Oct 25 2022, 4:03 PM.

Details

Reviewers

anlambert

Group Reviewers

Reviewers

Maniphest Tasks

T4656: AttributeError: 'NoneType' object has no attribute 'endswith'

Commits

rDCIDXa66d5b240ab7: metadata_dictionary: Systematically check input URLs before adding to graph

Summary

This is hopefully the definitive workaround for the PyLD issue.

Closes T4656

Diff Detail

Repository

rDCIDX Metadata indexer

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 32575
Build 51028: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 51027: arc lint + arc unit

Event Timeline

vlorentz created this revision.Oct 25 2022, 4:03 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptOct 25 2022, 4:03 PM

Build has FAILED

Patch application report for D8772 (id=31621)

Rebasing onto a51cbf3965...

Current branch diff-target is up to date.

Changes applied before test

commit 4148da3ef7f6b7a5b98151aa6179502842f215ba
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Oct 25 16:02:16 2022 +0200

    metadata_dictionary: Systematically check input URLs before adding to graph
    
    This is hopefully the definitive workaround for the PyLD issue.

Link to build: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/517/
See console output for more information: https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/517/console

Harbormaster returned this revision to the author for changes because remote builds failed.Oct 25 2022, 4:07 PM

Harbormaster failed remote builds in B32574: Diff 31621!

flake8 lied

Build is green

Patch application report for D8772 (id=31622)

Rebasing onto a51cbf3965...

Current branch diff-target is up to date.

Changes applied before test

commit a66d5b240ab77e6d8d1b9accf43d571489a3f7f0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue Oct 25 16:02:16 2022 +0200

    metadata_dictionary: Systematically check input URLs before adding to graph
    
    This is hopefully the definitive workaround for the PyLD issue.

See https://jenkins.softwareheritage.org/job/DCIDX/job/tests-on-diff/518/ for more details.

Harbormaster completed remote builds in B32575: Diff 31622.Oct 25 2022, 4:26 PM

vlorentz requested review of this revision.Oct 25 2022, 4:27 PM

vlorentz added a child revision: D8778: codemeta: Fix incorrect output namespace for dates and URLs.Oct 26 2022, 2:20 PM

LGTM, added some nitpicks about typing as inline comments.

swh/indexer/metadata_dictionary/utils.py
80	Typing could be more precise here. url: Optional[str]
108–111	Could be merged into a single if block. if url is None or " " in url or not urllib.parse.urlparse(url).netloc: return