Page MenuHomeSoftware Heritage
Differential D3834

Add a --check-config option for cli commands
ClosedPublic
Actions

Authored by douardda on Aug 24 2020, 4:51 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDSTO4532a4dc64e7: Add a --check-config option for cli commands
rDSTO2a35c0bb231c: Remove the deprecated config-path option from `swh storage rpc-serve` command
Summary

this allows to specify on command line wether to check the configuration
and read or write access to the storage at startup time (especially for
rpc-server and replay commands.)

Warning: this option defaults to "read" for the backfill command and
"write" for rpc-serve and replay commands, so now the creation of
the Storage instance used in cli commands *will be checked*.

Also remove the deprecated config-path option from swh storage rpc-serve command and include the validation of the presence of a "storage" config section
in the main storage click.group, where the config-file is actually
parsed.

Depends on D3832.

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Aug 24 2020, 4:51 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 24 2020, 4:51 PM
douardda updated this revision to Diff 13501.Aug 24 2020, 5:01 PM

rebase + add closes T2525 in the ci msg

swh-public-ci added a comment.Aug 24 2020, 5:16 PM

Build is green

Patch application report for D3834 (id=13496)

Could not rebase; Attempt merge onto f570f9336c...

Merge made by the 'recursive' strategy.
 swh/storage/__init__.py              |  12 ++-
 swh/storage/cli.py                   |  59 +++++++----
 swh/storage/postgresql/db.py         |  12 +++
 swh/storage/postgresql/storage.py    |   3 +
 swh/storage/pytest_plugin.py         |   3 +-
 swh/storage/tests/test_init.py       | 189 +++++++++++++++++++++++++++++------
 swh/storage/tests/test_postgresql.py |  19 ++++
 7 files changed, 243 insertions(+), 54 deletions(-)
Changes applied before test
commit 0fc87dcdc697634d5357d03a997436f1aa7d95ce
Merge: f570f933 c8fe81c3
Author: Jenkins user <jenkins@localhost>
Date:   Mon Aug 24 15:12:14 2020 +0000

    Merge branch 'diff-target' into HEAD

commit c8fe81c350256721162ed56573dbf1b8ee666be5
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:14:47 2020 +0200

    Add a --check-config option for cli commands
    
    this allows to specify on command line wether to check the configuration
    and read or write access to the storage at startup time (especially for
    `rpc-server` and `replay` commands.)
    
    Warning: this option defaults to "read" for the backfill command and
    "write" for rpc-serve and replay commands, so now the creation of
    the Storage instance used in cli commands *will be checked*.

commit b42f3bac0afd08b5bbfd2605b6de077bbed71d6e
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:10:30 2020 +0200

    Remove the deprecated config-path option from `swh storage rpc-serve` command
    
    include the validation of the presence of a "storage" config section
    in the main `storage` click.group, where the config-file is actually
    parsed.

commit ee1ed57625cc6db56efaff1e1051a9e412ccf135
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 17:39:59 2020 +0200

    Add support for a new "check_config" config option in get_storage()
    
    if "check_config" is present in get_storage()'s kwargs, call the
    storage.check_config() method and raise an exception is the result
    fails. The storage.check_config() method is called with the content
    of the "check_config" config value (so expected value for this
    currently is `check_config={"check_write": <bool>}`).

commit f7434a8cb9c1ecd72bc716ed6dd3cf5f05ad6893
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 15:43:35 2020 +0200

    Check for db version mismatch in PgStorage.check_config()

commit d0052506f4deb165dd06d28f415787e2cd31c8ee
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 12:00:37 2020 +0200

    Add a check_dbversion() method to the Db class
    
    This method compares the version stored in the database in the `dbversion`
    table with the currently declared version.
    
    This current version is declared as a simple `current_version` attribute
    on the Db class. **It must be updated jointly in 30-swh-schema.sql** when
    modifying the db schema. Note that if this is forgotten, the added test
    `test_dbversion` should fail.
    
    Related to T2525.

commit 346ad9c4197e57d377fed351b2353e14af486995
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 11:55:56 2020 +0200

    Fix pytest_plugin's database janitor: do not truncate the dbversion table

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/858/ for more details.

Harbormaster completed remote builds in B14756: Diff 13496.Aug 24 2020, 5:16 PM
swh-public-ci added a comment.Aug 24 2020, 5:34 PM

Build is green

Patch application report for D3834 (id=13501)

Rebasing onto cc33dd3596...

Current branch diff-target is up to date.
Changes applied before test
commit bc4f8ef0f647969d51481d77a8bcc8bbce70b416
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:14:47 2020 +0200

    Add a --check-config option for cli commands
    
    this allows to specify on command line wether to check the configuration
    and read or write access to the storage at startup time (especially for
    `rpc-server` and `replay` commands.)
    
    Warning: this option defaults to "read" for the backfill command and
    "write" for rpc-serve and replay commands, so now the creation of
    the Storage instance used in cli commands *will be checked*.
    
    Closes T2525.

commit f0fc833e8ef2ab12fec0340b2f5e4198feb1492b
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:10:30 2020 +0200

    Remove the deprecated config-path option from `swh storage rpc-serve` command
    
    include the validation of the presence of a "storage" config section
    in the main `storage` click.group, where the config-file is actually
    parsed.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/862/ for more details.

Harbormaster completed remote builds in B14761: Diff 13501.Aug 24 2020, 5:34 PM
ardumont accepted this revision.Aug 24 2020, 5:47 PM
ardumont added a subscriber: ardumont.

missing types on helper function but looks good

thanks.

swh/storage/cli.py
23

Could you please add types?

i gather dict, str, str ;)

This revision is now accepted and ready to land.Aug 24 2020, 5:47 PM
douardda updated this revision to Diff 13510.Aug 25 2020, 9:52 AM

type annotate the new ensure_check_config helper function

and add a docstring.

swh-public-ci added a comment.Aug 25 2020, 9:57 AM

Build is green

Patch application report for D3834 (id=13510)

Rebasing onto f3870dcd07...

First, rewinding head to replay your work on top of it...
Applying: Remove the deprecated config-path option from `swh storage rpc-serve` command
Applying: Add a --check-config option for cli commands
Changes applied before test
commit 47dae5cf9eef08816fa64c743bcfb322d36a9b6d
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:14:47 2020 +0200

    Add a --check-config option for cli commands
    
    this allows to specify on command line wether to check the configuration
    and read or write access to the storage at startup time (especially for
    `rpc-server` and `replay` commands.)
    
    Warning: this option defaults to "read" for the backfill command and
    "write" for rpc-serve and replay commands, so now the creation of
    the Storage instance used in cli commands *will be checked*.
    
    Closes T2525.

commit d3f8e24e535ded09a451fb661acf637f161bc65b
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:10:30 2020 +0200

    Remove the deprecated config-path option from `swh storage rpc-serve` command
    
    include the validation of the presence of a "storage" config section
    in the main `storage` click.group, where the config-file is actually
    parsed.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/866/ for more details.

Harbormaster completed remote builds in B14770: Diff 13510.Aug 25 2020, 9:57 AM
douardda updated this revision to Diff 13515.Aug 25 2020, 10:51 AM

rebase

swh-public-ci added a comment.Aug 25 2020, 10:56 AM

Build is green

Patch application report for D3834 (id=13515)

Rebasing onto e8b1b21652...

Current branch diff-target is up to date.
Changes applied before test
commit 4532a4dc64e7a1967e501303d00e6dfbce739ca7
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:14:47 2020 +0200

    Add a --check-config option for cli commands
    
    this allows to specify on command line wether to check the configuration
    and read or write access to the storage at startup time (especially for
    `rpc-server` and `replay` commands.)
    
    Warning: this option defaults to "read" for the backfill command and
    "write" for rpc-serve and replay commands, so now the creation of
    the Storage instance used in cli commands *will be checked*.
    
    Closes T2525.

commit 2a35c0bb231c177824f93c56c9d11bc24215a6ce
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Aug 24 16:10:30 2020 +0200

    Remove the deprecated config-path option from `swh storage rpc-serve` command
    
    include the validation of the presence of a "storage" config section
    in the main `storage` click.group, where the config-file is actually
    parsed.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/868/ for more details.

Harbormaster completed remote builds in B14792: Diff 13515.Aug 25 2020, 10:56 AM
Closed by commit rDSTO2a35c0bb231c: Remove the deprecated config-path option from `swh storage rpc-serve` command (authored by douardda). · Explain WhyAug 25 2020, 11:52 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDSTO2a35c0bb231c: Remove the deprecated config-path option from `swh storage rpc-serve` command.
douardda added a commit: rDSTO4532a4dc64e7: Add a --check-config option for cli commands.

Revision Contents
Changeset List

PathSize
swh/
storage/
30 lines

Diff 13518

View Options

swh/storage/cli.py

Loading...
About · Developer information · Legal