debian: Fix a couple of issues in the loader
ClosedPublic
Actions

Authored by anlambert on Dec 3 2021, 11:18 AM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T2400: Ingest current and historical Ubuntu releases

Commits

rDLDBASE22478fa8a0a4: package/debian: Handle extra sha1 sum in source package metadata
rDLDBASE3e675d0d70c5: debian/tasks: Rename loading task function to fix scheduling
rDLDBASEb423b682df36: debian: Remove unused date parameter of DebianLoader

Summary

That diff compiles a couple of fixes in order to make the loading of
debian packages functional again:

remove unused date parameter of DebianLoader

handle extra sha1 sum in source package metadata

rename loading task function to fix scheduling

See commits log for more details.

Related to T2400

Diff Detail

Repository

rDLDBASE Generic VCS/Package Loader

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Dec 3 2021, 11:18 AM

Herald added a reviewer: Reviewers. · View Herald TranscriptDec 3 2021, 11:18 AM

anlambert mentioned this in D6738: debian: Update extra_loader_arguments dict produced ListedOrigin models.Dec 3 2021, 11:19 AM

Build has FAILED

Patch application report for D6737 (id=24468)

Rebasing onto 9949093332...

Current branch diff-target is up to date.

Changes applied before test

commit f49d66d4f2f2bf952d99f1b3aaae6685d9b5376c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Dec 3 11:04:58 2021 +0100

    debian/tasks: Rename loading task function to fix scheduling
    
    Loading task function must be named load_{visit_type} in order for
    the scheduler to sucessfully create loading tasks.
    
    Visit type name for debian packages is deb so the loading task
    function must be renamed to load_deb.
    
    Related to T2400

commit 22478fa8a0a49e8187f7bb0454b24406777a5307
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Dec 2 20:03:13 2021 +0100

    package/debian: Handle extra sha1 sum in source package metadata
    
    Some debian source package metadata have extra sha1 sums for their
    files, for instance those from the ubuntu hirsute suite.
    
    So add an optional sha1 field in DebianFileMetadata model in order
    to avoid loading errors.
    
    Related to T2400

commit b423b682df36882e48a9656bdf8c41968475dae6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Dec 2 15:20:29 2021 +0100

    debian: Remove unused date parameter of DebianLoader

Link to build: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/637/
See console output for more information: https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/637/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 3 2021, 11:20 AM

Harbormaster failed remote builds in B25341: Diff 24468!

Fix CLI test

Build is green

Patch application report for D6737 (id=24470)

Rebasing onto 9949093332...

Current branch diff-target is up to date.

Changes applied before test

commit 3e675d0d70c5f27fb0998a1458cb0abcdc878bec
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Fri Dec 3 11:04:58 2021 +0100

    debian/tasks: Rename loading task function to fix scheduling
    
    Loading task function must be named load_{visit_type} in order for
    the scheduler to sucessfully create loading tasks.
    
    Visit type name for debian packages is deb so the loading task
    function must be renamed to load_deb.
    
    Related to T2400

commit 22478fa8a0a49e8187f7bb0454b24406777a5307
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Dec 2 20:03:13 2021 +0100

    package/debian: Handle extra sha1 sum in source package metadata
    
    Some debian source package metadata have extra sha1 sums for their
    files, for instance those from the ubuntu hirsute suite.
    
    So add an optional sha1 field in DebianFileMetadata model in order
    to avoid loading errors.
    
    Related to T2400

commit b423b682df36882e48a9656bdf8c41968475dae6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Dec 2 15:20:29 2021 +0100

    debian: Remove unused date parameter of DebianLoader

See https://jenkins.softwareheritage.org/job/DLDBASE/job/tests-on-diff/638/ for more details.

Harbormaster completed remote builds in B25343: Diff 24470.Dec 3 2021, 11:24 AM

anlambert requested review of this revision.Dec 3 2021, 11:24 AM