tasks: Fix and simplify implementation
ClosedPublic
Actions

Authored by anlambert on Apr 26 2022, 5:16 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T4187: Pass forge type to loaders

Commits

rDLDSVNb4bca94f058b: tasks: Fix and simplify implementation

Summary

Recent changes in swh-scheduler add new parameters to the celery tasks
produced from swh.scheduler.model.ListedOrigin instances.

Those new parameters were not properly handled in the celery tasks
implementation for svn loading but also in some svn loader ones.

So ensure to handle any new parameters by not hardcoding the expected
ones in task signatures but also allowing any extra keyword parameters
in loader constructors.

Also remove explicit setting of visit_date if none has been provided
as task parameter as it is already handled in loader constructor.

Related to T4187

Diff Detail

Repository

rDLDSVN Subversion (SVN) loader

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Apr 26 2022, 5:16 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptApr 26 2022, 5:16 PM

Build is green

Patch application report for D7690 (id=27787)

Rebasing onto 91dc7ad9d8...

Current branch diff-target is up to date.

Changes applied before test

commit 111e8fd7260d3dc9dfb553837141149686e1bd4d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Apr 26 17:00:27 2022 +0200

    tasks: Fix and simplify implementation
    
    Recent changes in swh-scheduler add new parameters to the celery tasks
    produced from swh.scheduler.model.ListedOrigin instances.
    
    Those new parameters were not properly handled in the celery tasks
    implementation for svn loading but also in some svn loader ones.
    
    So ensure to handle any new parameters by not hardcoding the expected
    ones in task signatures but also allowing any extra keyword parameters
    in loader constructors.
    
    Also remove explicit setting of visit_date if none has been provided
    as task parameter as it is already handled in loader constructor.
    
    Related to T4187

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/324/ for more details.

Harbormaster completed remote builds in B28848: Diff 27787.Apr 26 2022, 5:19 PM

anlambert requested review of this revision.Apr 26 2022, 5:19 PM

lgtm

one comment inline.

swh/loader/svn/loader.py
565	i think this one can go away as well now (same for the next one). (It's dealt with in the base loader class).

ardumont accepted this revision.Apr 26 2022, 5:26 PM

This revision is now accepted and ready to land.Apr 26 2022, 5:26 PM

Update:

Remove max_content_size parameter to loaders (already handled by BaseLoader class)
Ensure to pass loader extra keyword parameters to base class constructor

anlambert added inline comments.Apr 26 2022, 5:41 PM

swh/loader/svn/loader.py
565	Right and I also forgot to forward the extra keyword parameters to base constructor.

Build is green

Patch application report for D7690 (id=27789)

Rebasing onto 91dc7ad9d8...

Current branch diff-target is up to date.

Changes applied before test

commit b4bca94f058b0cda11c257d22567343b26ed52b6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Apr 26 17:00:27 2022 +0200

    tasks: Fix and simplify implementation
    
    Recent changes in swh-scheduler add new parameters to the celery tasks
    produced from swh.scheduler.model.ListedOrigin instances.
    
    Those new parameters were not properly handled in the celery tasks
    implementation for svn loading but also in some svn loader ones.
    
    So ensure to handle any new parameters by not hardcoding the expected
    ones in task signatures but also allowing any extra keyword parameters
    in loader constructors.
    
    Also remove explicit setting of visit_date if none has been provided
    as task parameter as it is already handled in loader constructor.
    
    Related to T4187

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/325/ for more details.

Harbormaster completed remote builds in B28851: Diff 27789.Apr 26 2022, 5:43 PM

Closed by commit rDLDSVNb4bca94f058b: tasks: Fix and simplify implementation (authored by anlambert). · Explain WhyApr 27 2022, 10:43 AM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDLDSVNb4bca94f058b: tasks: Fix and simplify implementation.