debian: Fix a couple of issues in the loader
ClosedPublic
Actions

Authored by anlambert on Dec 3 2021, 11:18 AM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T2400: Ingest current and historical Ubuntu releases

Commits

rDLDBASE22478fa8a0a4: package/debian: Handle extra sha1 sum in source package metadata
rDLDBASE3e675d0d70c5: debian/tasks: Rename loading task function to fix scheduling
rDLDBASEb423b682df36: debian: Remove unused date parameter of DebianLoader

Summary

That diff compiles a couple of fixes in order to make the loading of
debian packages functional again:

remove unused date parameter of DebianLoader

handle extra sha1 sum in source package metadata

rename loading task function to fix scheduling

See commits log for more details.

Related to T2400

Diff Detail

Repository

rDLDBASE Generic VCS/Package Loader

Branch

debian-loader-fixes

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 25341
Build 39611: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 39610: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	30 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.tests.test_cli::test_get_loader swh_loader_config = {'deposit': {'auth': {'password': 'pass', 'username': 'user'}, 'url': 'https://deposit.softwareheritage.org/1/private'}, 'storage': {'cls': 'memory'}} def test_get_loader(swh_loader_config):
	1 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.core.tests.test_converters::test_content_for_storage_data
	4 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.core.tests.test_converters::test_content_for_storage_path
	1 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.core.tests.test_converters::test_content_for_storage_too_long
	1 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.loader.core.tests.test_converters::test_prepare_contents
		View Full Test Results (1 Failed · 166 Passed)

Event Timeline

anlambert created this revision.Dec 3 2021, 11:18 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptDec 3 2021, 11:18 AM

anlambert mentioned this in D6738: debian: Update extra_loader_arguments dict produced ListedOrigin models.Dec 3 2021, 11:19 AM

Build has FAILED

Patch application report for D6737 (id=24468)

Rebasing onto 9949093332...

Current branch diff-target is up to date.

Changes applied before test

commit f49d66d4f2f2bf952d99f1b3aaae6685d9b5376c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Dec 3 11:04:58 2021 +0100

    debian/tasks: Rename loading task function to fix scheduling
    
    Loading task function must be named load_{visit_type} in order for
    the scheduler to sucessfully create loading tasks.
    
    Visit type name for debian packages is deb so the loading task
    function must be renamed to load_deb.
    
    Related to T2400

commit 22478fa8a0a49e8187f7bb0454b24406777a5307
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Dec 2 20:03:13 2021 +0100

    package/debian: Handle extra sha1 sum in source package metadata
    
    Some debian source package metadata have extra sha1 sums for their
    files, for instance those from the ubuntu hirsute suite.
    
    So add an optional sha1 field in DebianFileMetadata model in order
    to avoid loading errors.
    
    Related to T2400

commit b423b682df36882e48a9656bdf8c41968475dae6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Dec 2 15:20:29 2021 +0100

    debian: Remove unused date parameter of DebianLoader

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/637/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/637/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 3 2021, 11:20 AM

Harbormaster failed remote builds in B25341: Diff 24468!

Fix CLI test

Build is green

Patch application report for D6737 (id=24470)

Rebasing onto 9949093332...

Current branch diff-target is up to date.

Changes applied before test

commit 3e675d0d70c5f27fb0998a1458cb0abcdc878bec
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Dec 3 11:04:58 2021 +0100

    debian/tasks: Rename loading task function to fix scheduling
    
    Loading task function must be named load_{visit_type} in order for
    the scheduler to sucessfully create loading tasks.
    
    Visit type name for debian packages is deb so the loading task
    function must be renamed to load_deb.
    
    Related to T2400

commit 22478fa8a0a49e8187f7bb0454b24406777a5307
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Dec 2 20:03:13 2021 +0100

    package/debian: Handle extra sha1 sum in source package metadata
    
    Some debian source package metadata have extra sha1 sums for their
    files, for instance those from the ubuntu hirsute suite.
    
    So add an optional sha1 field in DebianFileMetadata model in order
    to avoid loading errors.
    
    Related to T2400

commit b423b682df36882e48a9656bdf8c41968475dae6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Dec 2 15:20:29 2021 +0100

    debian: Remove unused date parameter of DebianLoader

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/638/ for more details.

Harbormaster completed remote builds in B25343: Diff 24470.Dec 3 2021, 11:24 AM

anlambert requested review of this revision.Dec 3 2021, 11:24 AM