Page MenuHomeSoftware Heritage

tasks: Fix and simplify implementation
ClosedPublic

Authored by anlambert on Apr 26 2022, 5:16 PM.

Details

Summary

Recent changes in swh-scheduler add new parameters to the celery tasks
produced from swh.scheduler.model.ListedOrigin instances.

Those new parameters were not properly handled in the celery tasks
implementation for svn loading but also in some svn loader ones.

So ensure to handle any new parameters by not hardcoding the expected
ones in task signatures but also allowing any extra keyword parameters
in loader constructors.

Also remove explicit setting of visit_date if none has been provided
as task parameter as it is already handled in loader constructor.

Related to T4187

Diff Detail

Repository
rDLDSVN Subversion (SVN) loader
Branch
simplify-tasks-implementation
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 28851
Build 45087: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 45086: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D7690 (id=27787)

Rebasing onto 91dc7ad9d8...

Current branch diff-target is up to date.
Changes applied before test
commit 111e8fd7260d3dc9dfb553837141149686e1bd4d
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Apr 26 17:00:27 2022 +0200

    tasks: Fix and simplify implementation
    
    Recent changes in swh-scheduler add new parameters to the celery tasks
    produced from swh.scheduler.model.ListedOrigin instances.
    
    Those new parameters were not properly handled in the celery tasks
    implementation for svn loading but also in some svn loader ones.
    
    So ensure to handle any new parameters by not hardcoding the expected
    ones in task signatures but also allowing any extra keyword parameters
    in loader constructors.
    
    Also remove explicit setting of visit_date if none has been provided
    as task parameter as it is already handled in loader constructor.
    
    Related to T4187

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/324/ for more details.

lgtm

one comment inline.

swh/loader/svn/loader.py
565

i think this one can go away as well now (same for the next one).
(It's dealt with in the base loader class).

This revision is now accepted and ready to land.Apr 26 2022, 5:26 PM

Update:

  • Remove max_content_size parameter to loaders (already handled by BaseLoader class)
  • Ensure to pass loader extra keyword parameters to base class constructor
swh/loader/svn/loader.py
565

Right and I also forgot to forward the extra keyword parameters to base constructor.

Build is green

Patch application report for D7690 (id=27789)

Rebasing onto 91dc7ad9d8...

Current branch diff-target is up to date.
Changes applied before test
commit b4bca94f058b0cda11c257d22567343b26ed52b6
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Apr 26 17:00:27 2022 +0200

    tasks: Fix and simplify implementation
    
    Recent changes in swh-scheduler add new parameters to the celery tasks
    produced from swh.scheduler.model.ListedOrigin instances.
    
    Those new parameters were not properly handled in the celery tasks
    implementation for svn loading but also in some svn loader ones.
    
    So ensure to handle any new parameters by not hardcoding the expected
    ones in task signatures but also allowing any extra keyword parameters
    in loader constructors.
    
    Also remove explicit setting of visit_date if none has been provided
    as task parameter as it is already handled in loader constructor.
    
    Related to T4187

See https://jenkins.softwareheritage.org/job/DLDSVN/job/tests-on-diff/325/ for more details.