Page MenuHomeSoftware Heritage
Differential D3832

Add support for a new "check_config" config option in get_storage()
ClosedPublic
Actions

Authored by douardda on Aug 24 2020, 2:51 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T2525: Prevent the storage server from starting in case of db version mismatch
Commits
rDSTOcc33dd359692: Add support for a new "check_config" config option in get_storage()
Summary

if "check_config" is present in get_storage()'s kwargs, call the
storage.check_config() method and raise an exception is the result
fails. The storage.check_config() method is called with the content
of the "check_config" config value (so expected value for this
currently is check_config={"check_write": <bool>}).

Depends on D3831.
Related to T2525.

Diff Detail

Event Timeline

douardda created this revision.Aug 24 2020, 2:51 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 24 2020, 2:51 PM
swh-public-ci added a comment.Aug 24 2020, 2:59 PM

Build is green

Patch application report for D3832 (id=13486)

Could not rebase; Attempt merge onto d1f19e97c3...

Updating d1f19e97..589cdcb7
Fast-forward
 swh/storage/__init__.py              |  12 ++-
 swh/storage/postgresql/db.py         |  12 +++
 swh/storage/postgresql/storage.py    |   3 +
 swh/storage/pytest_plugin.py         |   3 +-
 swh/storage/tests/test_init.py       | 185 +++++++++++++++++++++++++++++------
 swh/storage/tests/test_postgresql.py |  19 ++++
 6 files changed, 202 insertions(+), 32 deletions(-)
Changes applied before test
commit 589cdcb73ac4c7c8246846bec0017931cd2e113b
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 17:39:59 2020 +0200

    Add support for a new "check_config" config option in get_storage()
    
    if "check_config" is present in get_storage()'s kwargs, call the
    storage.check_config() method and raise an exception is the result
    fails. The storage.check_config() method is called with the content
    of the "check_config" config value (so expected value for this
    currently is `check_config={"check_write": <bool>}`).

commit 0dbe0e644dbad7cd0b1f30b1370682fe61558fde
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 15:43:35 2020 +0200

    Check for db version mismatch in PgStorage.check_config()

commit cba6959b163ce0711b3b490201dd6f414843396e
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 12:00:37 2020 +0200

    Add a check_dbversion() method to the Db class
    
    This method compares the version stored in the database in the `dbversion`
    table with the currently declared version.
    
    This current version is declared as a simple `current_version` attribute
    on the Db class. **It must be updated jointly in 30-swh-schema.sql** when
    modifying the db schema. Note that if this is forgotten, the added test
    `test_dbversion` should fail.
    
    Related to T2525.

commit 346ad9c4197e57d377fed351b2353e14af486995
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 11:55:56 2020 +0200

    Fix pytest_plugin's database janitor: do not truncate the dbversion table

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/849/ for more details.

Harbormaster completed remote builds in B14746: Diff 13486.Aug 24 2020, 2:59 PM
ardumont added a subscriber: ardumont.Aug 24 2020, 3:47 PM
ardumont added inline comments.
swh/storage/tests/test_init.py
184

"factory"

186

and checks (drop and that).

ardumont accepted this revision.Aug 24 2020, 3:49 PM

2 small typos but otherwise lgtm ;)

This revision is now accepted and ready to land.Aug 24 2020, 3:49 PM
douardda updated this revision to Diff 13491.Aug 24 2020, 4:41 PM

rebase

douardda updated this revision to Diff 13495.Aug 24 2020, 4:49 PM

rebase

douardda added a child revision: D3834: Add a --check-config option for cli commands.Aug 24 2020, 4:51 PM
swh-public-ci added a comment.Aug 24 2020, 4:54 PM

Build is green

Patch application report for D3832 (id=13491)

Could not rebase; Attempt merge onto f570f9336c...

Merge made by the 'recursive' strategy.
 swh/storage/__init__.py              |  12 ++-
 swh/storage/postgresql/db.py         |  12 +++
 swh/storage/postgresql/storage.py    |   3 +
 swh/storage/pytest_plugin.py         |   3 +-
 swh/storage/tests/test_cli.py        |  10 +-
 swh/storage/tests/test_init.py       | 189 +++++++++++++++++++++++++++++------
 swh/storage/tests/test_postgresql.py |  19 ++++
 7 files changed, 211 insertions(+), 37 deletions(-)
Changes applied before test
commit 73a99e260be4ec36b99b84fb492dd7ddb6850ff3
Merge: f570f933 bf011bd6
Author: Jenkins user <jenkins@localhost>
Date:   Mon Aug 24 14:49:42 2020 +0000

    Merge branch 'diff-target' into HEAD

commit bf011bd6483815cafd6c6df97ae4f5ec4f0cb4c3
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 17:39:59 2020 +0200

    Add support for a new "check_config" config option in get_storage()
    
    if "check_config" is present in get_storage()'s kwargs, call the
    storage.check_config() method and raise an exception is the result
    fails. The storage.check_config() method is called with the content
    of the "check_config" config value (so expected value for this
    currently is `check_config={"check_write": <bool>}`).

commit c90c35826589b64e9e4761dd51c623a7d1245c32
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 15:43:35 2020 +0200

    Check for db version mismatch in PgStorage.check_config()

commit 80e756b0662108f94e64eb7eccf3a6b2ad4a61b7
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 12:00:37 2020 +0200

    Add a check_dbversion() method to the Db class
    
    This method compares the version stored in the database in the `dbversion`
    table with the currently declared version.
    
    This current version is declared as a simple `current_version` attribute
    on the Db class. **It must be updated jointly in 30-swh-schema.sql** when
    modifying the db schema. Note that if this is forgotten, the added test
    `test_dbversion` should fail.
    
    Related to T2525.

commit 346ad9c4197e57d377fed351b2353e14af486995
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 11:55:56 2020 +0200

    Fix pytest_plugin's database janitor: do not truncate the dbversion table

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/853/ for more details.

Harbormaster completed remote builds in B14751: Diff 13491.Aug 24 2020, 4:54 PM
douardda updated this revision to Diff 13500.Aug 24 2020, 5:00 PM

rebase

swh-public-ci added a comment.Aug 24 2020, 5:12 PM

Build is green

Patch application report for D3832 (id=13495)

Could not rebase; Attempt merge onto f570f9336c...

Merge made by the 'recursive' strategy.
 swh/storage/__init__.py              |  12 ++-
 swh/storage/postgresql/db.py         |  12 +++
 swh/storage/postgresql/storage.py    |   3 +
 swh/storage/pytest_plugin.py         |   3 +-
 swh/storage/tests/test_init.py       | 189 +++++++++++++++++++++++++++++------
 swh/storage/tests/test_postgresql.py |  19 ++++
 6 files changed, 206 insertions(+), 32 deletions(-)
Changes applied before test
commit 816f1a28b6c2afca5e1e41347cd6a6a78b1c520d
Merge: f570f933 ee1ed576
Author: Jenkins user <jenkins@localhost>
Date:   Mon Aug 24 15:07:55 2020 +0000

    Merge branch 'diff-target' into HEAD

commit ee1ed57625cc6db56efaff1e1051a9e412ccf135
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 17:39:59 2020 +0200

    Add support for a new "check_config" config option in get_storage()
    
    if "check_config" is present in get_storage()'s kwargs, call the
    storage.check_config() method and raise an exception is the result
    fails. The storage.check_config() method is called with the content
    of the "check_config" config value (so expected value for this
    currently is `check_config={"check_write": <bool>}`).

commit f7434a8cb9c1ecd72bc716ed6dd3cf5f05ad6893
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 15:43:35 2020 +0200

    Check for db version mismatch in PgStorage.check_config()

commit d0052506f4deb165dd06d28f415787e2cd31c8ee
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 12:00:37 2020 +0200

    Add a check_dbversion() method to the Db class
    
    This method compares the version stored in the database in the `dbversion`
    table with the currently declared version.
    
    This current version is declared as a simple `current_version` attribute
    on the Db class. **It must be updated jointly in 30-swh-schema.sql** when
    modifying the db schema. Note that if this is forgotten, the added test
    `test_dbversion` should fail.
    
    Related to T2525.

commit 346ad9c4197e57d377fed351b2353e14af486995
Author: David Douard <david.douard@sdfa3.org>
Date:   Thu Aug 20 11:55:56 2020 +0200

    Fix pytest_plugin's database janitor: do not truncate the dbversion table

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/857/ for more details.

Harbormaster completed remote builds in B14755: Diff 13495.Aug 24 2020, 5:12 PM
This revision was landed with ongoing or failed builds.Aug 24 2020, 5:24 PM
Closed by commit rDSTOcc33dd359692: Add support for a new "check_config" config option in get_storage() (authored by douardda). · Explain Why
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDSTOcc33dd359692: Add support for a new "check_config" config option in get_storage().
swh-public-ci added a comment.Aug 24 2020, 5:30 PM

Build is green

Patch application report for D3832 (id=13500)

Rebasing onto cc33dd3596...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-861-D3832.
Changes applied before test



See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/861/ for more details.
Harbormaster completed remote builds in B14760: Diff 13500.Aug 24 2020, 5:30 PM
Revision Contents
Changeset List
PathSizeCoverage (All)Coverage (Touched)
swh/
storage/
12 lines97%
Loading...
1 line98%
Loading...
tests/
189 lines100%
Loading...
Diff 13495
View Options
swh/storage/__init__.py
Loading...
View Options
swh/storage/pytest_plugin.py
Loading...
View Options
swh/storage/tests/test_init.py
Loading...
About · Developer information · Legal