Page MenuHomeSoftware Heritage

pattern: Fix and improve config overriding in from_configfile method
ClosedPublic

Authored by anlambert on Jan 15 2021, 2:54 PM.

Details

Summary

Fix error when a configuration value loaded from a config file is also
given as keyword parameter to the from_configfile method.

Override configuration loaded from config file only if the provided
value is not None.

The purpose of the second change is to be able to write a task implementation
the following way (I take the Bitbucket lister as an example):

@shared_task(name=__name__ + ".IncrementalBitBucketLister")
def list_bitbucket_incremental(
    page_size: Optional[int] = None,
    username: Optional[str] = None,
    password: Optional[str] = None,
):
    """Incremental listing of the public Bitbucket repositories."""
    lister = BitbucketLister.from_configfile(
	page_size=page_size, username=username, password=pasword)
    return lister.run().dict()

When all parameter values are None, those from the config file will
then be used.

When one of a task parameter is not None, it will override the value
from the config file. This is useful when testing a lister in the docker
environment as parameters can be overridden through the swh scheduler CLI.

Diff Detail

Repository
rDLS Listers
Branch
pattern-from-config-file-fix
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 18453
Build 28530: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 28529: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D4872 (id=17276)

Rebasing onto c782275296...

Current branch diff-target is up to date.
Changes applied before test
commit 403a158751a191039b0b37a32fefee7184053053
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Jan 15 14:39:41 2021 +0100

    pattern: Fix and improve config overriding in from_configfile method
    
    Fix error when a configuration value loaded from a config file is also
    given as keyword parameter to the from_configfile method.
    
    Override configuration loaded from config file only if the provided
    value is not None.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/107/ for more details.

This revision is now accepted and ready to land.Jan 18 2021, 10:56 AM

Makes sense, thanks!

Could you document this behavior in the from_configfile docstring (just add "if they're not None" in the description of kwargs)?

Rebase and update docstring

Build is green

Patch application report for D4872 (id=17314)

Rebasing onto a41c03e4c8...

Current branch diff-target is up to date.
Changes applied before test
commit 9fd91f007d0530ffcb7bbe4e4a8fb006fbe39396
Author: Antoine Lambert <antoine.lambert@inria.fr>
Date:   Fri Jan 15 14:39:41 2021 +0100

    pattern: Fix and improve config overriding in from_configfile method
    
    Fix error when a configuration value loaded from a config file is also
    given as keyword parameter to the from_configfile method.
    
    Override configuration loaded from config file only if the provided
    value is not None.

See https://jenkins.softwareheritage.org/job/DLS/job/tests-on-diff/115/ for more details.