Page MenuHomeSoftware Heritage

Prevent root conftest.py from loading swh.core at module loading time
ClosedPublic

Authored by douardda on Feb 16 2022, 1:25 PM.

Details

Summary

move the import statement in swhmain() body.
Having the swh.core module loaded with the execution of conftest.py can
have weird side effects, especially on pytest's discovery mechanism in
the context of tox, namely make pytest load test files from the source
directory rather than from the installed package in the tox venv,
potentially leading to ImportPathMismatchError.

Depends on D7159

Event Timeline

Build is green

Patch application report for D7189 (id=26037)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..72f2b90
Fast-forward
 MANIFEST.in                                        |   2 +
 conftest.py                                        |   4 +-
 docs/db.rst                                        | 181 +++++++++++++
 docs/index.rst                                     |   1 +
 swh/core/cli/db.py                                 | 200 +++++++++++++-
 swh/core/db/db_utils.py                            | 293 +++++++++++++++++++--
 swh/core/db/pytest_plugin.py                       | 213 ++++++++++-----
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  50 ++++
 .../tests/data/cli/{ => sql}/0-superuser-init.sql  |   0
 .../data/cli/{1-schema.sql => sql/30-schema.sql}   |   2 +-
 .../data/cli/{3-func.sql => sql/40-funcs.sql}      |   0
 .../tests/data/cli/{4-data.sql => sql/50-data.sql} |   2 +-
 .../db/tests/data/cli_new/sql/0-superuser-init.sql |   1 +
 swh/core/db/tests/data/cli_new/sql/30-schema.sql   |   6 +
 swh/core/db/tests/data/cli_new/sql/40-funcs.sql    |   6 +
 swh/core/db/tests/data/cli_new/sql/50-data.sql     |   2 +
 .../db/tests/data/cli_new/sql/upgrades/001.sql     |   5 +
 .../db/tests/data/cli_new/sql/upgrades/002.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/003.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/004.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/005.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/006.sql     |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 214 +++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 181 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 29 files changed, 1307 insertions(+), 148 deletions(-)
 create mode 100644 docs/db.rst
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%)
 rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 72f2b90b66694a72ca1dc082156a7954cc43a743
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 15 14:14:45 2022 +0100

    Prevent root conftest.py from loading swh.core at module loading time
    
    move the import statement in swhmain() body.
    Having the swh.core module loaded with the execution of conftest.py can
    have weird side effects, especially on pytest's discovery mechanism in
    the context of tox, namely make pytest load test files from the source
    directory rather than from the installed package in the tox venv,
    potentially leading to ImportPathMismatchError.

commit 33ae35845b07e2651cb407a1430802288dd56ad1
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 15:54:25 2022 +0100

    Add a documentation section for swh.core.db
    
    this is far from completion.

commit a06d92f6b608df9b057f27e2c59fc1b9785779a0
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 12:37:07 2022 +0100

    Make get_sql_for_package() return pathlib.Path objects
    
    and replace mock_package_sql by a mock_import_swhmodule fixture in favor
    of using the stock get_sql_for_package() function (on the mock module
    returned by mock_import_swhmodule).
    
    This also comes with a change in expected test data structure for sql
    script; the `tests/data/<mod>` is now expected to store the sql script
    files in a dedicated `sql/` sub directory instead of having them
    directly in the `<mod>` directory so that the get_sql_for_package() can
    be used in tests unmocked.

commit d187e7e4574bb190a3af8254ec116a89a59e34bf
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit dfa01610264a925ba02e33c6f2401d34fd34ec18
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/328/ for more details.

Build is green

Patch application report for D7189 (id=26039)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..2e88d92
Fast-forward
 MANIFEST.in                                        |   2 +
 conftest.py                                        |   4 +-
 docs/db.rst                                        | 176 +++++++++++++
 docs/index.rst                                     |   1 +
 swh/core/cli/db.py                                 | 200 +++++++++++++-
 swh/core/db/db_utils.py                            | 293 +++++++++++++++++++--
 swh/core/db/pytest_plugin.py                       | 213 ++++++++++-----
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  50 ++++
 .../tests/data/cli/{ => sql}/0-superuser-init.sql  |   0
 .../data/cli/{1-schema.sql => sql/30-schema.sql}   |   2 +-
 .../data/cli/{3-func.sql => sql/40-funcs.sql}      |   0
 .../tests/data/cli/{4-data.sql => sql/50-data.sql} |   2 +-
 .../db/tests/data/cli_new/sql/0-superuser-init.sql |   1 +
 swh/core/db/tests/data/cli_new/sql/30-schema.sql   |   6 +
 swh/core/db/tests/data/cli_new/sql/40-funcs.sql    |   6 +
 swh/core/db/tests/data/cli_new/sql/50-data.sql     |   2 +
 .../db/tests/data/cli_new/sql/upgrades/001.sql     |   5 +
 .../db/tests/data/cli_new/sql/upgrades/002.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/003.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/004.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/005.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/006.sql     |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 214 +++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 181 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 29 files changed, 1302 insertions(+), 148 deletions(-)
 create mode 100644 docs/db.rst
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%)
 rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 2e88d92a1ccf0e236faa18ce064a7e2db3998dfc
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 15 14:14:45 2022 +0100

    Prevent root conftest.py from loading swh.core at module loading time
    
    move the import statement in swhmain() body.
    Having the swh.core module loaded with the execution of conftest.py can
    have weird side effects, especially on pytest's discovery mechanism in
    the context of tox, namely make pytest load test files from the source
    directory rather than from the installed package in the tox venv,
    potentially leading to ImportPathMismatchError.

commit bb77dc02d7cbefc989561e292c88e53aecfa8100
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 15:54:25 2022 +0100

    Add a documentation section for swh.core.db
    
    this is far from completion.

commit a06d92f6b608df9b057f27e2c59fc1b9785779a0
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 12:37:07 2022 +0100

    Make get_sql_for_package() return pathlib.Path objects
    
    and replace mock_package_sql by a mock_import_swhmodule fixture in favor
    of using the stock get_sql_for_package() function (on the mock module
    returned by mock_import_swhmodule).
    
    This also comes with a change in expected test data structure for sql
    script; the `tests/data/<mod>` is now expected to store the sql script
    files in a dedicated `sql/` sub directory instead of having them
    directly in the `<mod>` directory so that the get_sql_for_package() can
    be used in tests unmocked.

commit d187e7e4574bb190a3af8254ec116a89a59e34bf
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit dfa01610264a925ba02e33c6f2401d34fd34ec18
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/330/ for more details.

Build is green

Patch application report for D7189 (id=26044)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..0bd342b
Fast-forward
 MANIFEST.in                                        |   2 +
 conftest.py                                        |   4 +-
 docs/db.rst                                        | 176 ++++++++++++
 docs/index.rst                                     |   1 +
 swh/core/cli/db.py                                 | 200 +++++++++++++-
 swh/core/db/db_utils.py                            | 298 +++++++++++++++++++--
 swh/core/db/pytest_plugin.py                       | 213 ++++++++++-----
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  50 ++++
 .../tests/data/cli/{ => sql}/0-superuser-init.sql  |   0
 .../data/cli/{1-schema.sql => sql/30-schema.sql}   |   2 +-
 .../data/cli/{3-func.sql => sql/40-funcs.sql}      |   0
 .../tests/data/cli/{4-data.sql => sql/50-data.sql} |   2 +-
 .../db/tests/data/cli_new/sql/0-superuser-init.sql |   1 +
 swh/core/db/tests/data/cli_new/sql/30-schema.sql   |   6 +
 swh/core/db/tests/data/cli_new/sql/40-funcs.sql    |   6 +
 swh/core/db/tests/data/cli_new/sql/50-data.sql     |   2 +
 .../db/tests/data/cli_new/sql/upgrades/001.sql     |   5 +
 .../db/tests/data/cli_new/sql/upgrades/002.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/003.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/004.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/005.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/006.sql     |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 214 +++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 181 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 29 files changed, 1307 insertions(+), 148 deletions(-)
 create mode 100644 docs/db.rst
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%)
 rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 0bd342bcd400bbfdb33468830fcc5dfe4d516811
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 15 14:14:45 2022 +0100

    Prevent root conftest.py from loading swh.core at module loading time
    
    move the import statement in swhmain() body.
    Having the swh.core module loaded with the execution of conftest.py can
    have weird side effects, especially on pytest's discovery mechanism in
    the context of tox, namely make pytest load test files from the source
    directory rather than from the installed package in the tox venv,
    potentially leading to ImportPathMismatchError.

commit 8646bc13577331bae62c6536dae05cd470be0295
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 15:54:25 2022 +0100

    Add a documentation section for swh.core.db
    
    this is far from completion.

commit ef48c771fe9ef7f797ecc49672e4aa470b1e3a1f
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 12:37:07 2022 +0100

    Make get_sql_for_package() return pathlib.Path objects
    
    and replace mock_package_sql by a mock_import_swhmodule fixture in favor
    of using the stock get_sql_for_package() function (on the mock module
    returned by mock_import_swhmodule).
    
    This also comes with a change in expected test data structure for sql
    script; the `tests/data/<mod>` is now expected to store the sql script
    files in a dedicated `sql/` sub directory instead of having them
    directly in the `<mod>` directory so that the get_sql_for_package() can
    be used in tests unmocked.

commit 9e599f03076f85b44ac0eb5d001ff6367a930276
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 1a187ca7787503d41ac13b2678b29f332d8fd200
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/335/ for more details.

ardumont added a subscriber: ardumont.

lgtm

It might be worth considering moving your git commit description as comment near the import statement.

This revision is now accepted and ready to land.Feb 16 2022, 2:18 PM

Build is green

Patch application report for D7189 (id=26048)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..e50d29e
Fast-forward
 MANIFEST.in                                        |   2 +
 conftest.py                                        |   4 +-
 docs/db.rst                                        | 176 ++++++++++++
 docs/index.rst                                     |   1 +
 swh/core/cli/db.py                                 | 200 +++++++++++++-
 swh/core/db/db_utils.py                            | 298 +++++++++++++++++++--
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++-----
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  50 ++++
 .../tests/data/cli/{ => sql}/0-superuser-init.sql  |   0
 .../data/cli/{1-schema.sql => sql/30-schema.sql}   |   2 +-
 .../data/cli/{3-func.sql => sql/40-funcs.sql}      |   0
 .../tests/data/cli/{4-data.sql => sql/50-data.sql} |   2 +-
 .../db/tests/data/cli_new/sql/0-superuser-init.sql |   1 +
 swh/core/db/tests/data/cli_new/sql/30-schema.sql   |   6 +
 swh/core/db/tests/data/cli_new/sql/40-funcs.sql    |   6 +
 swh/core/db/tests/data/cli_new/sql/50-data.sql     |   2 +
 .../db/tests/data/cli_new/sql/upgrades/001.sql     |   5 +
 .../db/tests/data/cli_new/sql/upgrades/002.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/003.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/004.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/005.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/006.sql     |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 214 +++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 181 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 29 files changed, 1323 insertions(+), 149 deletions(-)
 create mode 100644 docs/db.rst
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%)
 rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit e50d29eaec92c69967b34bf58fa2fa88b8842f2c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 15 14:14:45 2022 +0100

    Prevent root conftest.py from loading swh.core at module loading time
    
    move the import statement in swhmain() body.
    Having the swh.core module loaded with the execution of conftest.py can
    have weird side effects, especially on pytest's discovery mechanism in
    the context of tox, namely make pytest load test files from the source
    directory rather than from the installed package in the tox venv,
    potentially leading to ImportPathMismatchError.

commit 3cc57bb4e1d2dbacdc8a7958281a55cd469d01ca
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 15:54:25 2022 +0100

    Add a documentation section for swh.core.db
    
    this is far from completion.

commit 9e6f8fd036c86602be5f7c7d7df6c3836006aff5
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 12:37:07 2022 +0100

    Make get_sql_for_package() return pathlib.Path objects
    
    and replace mock_package_sql by a mock_import_swhmodule fixture in favor
    of using the stock get_sql_for_package() function (on the mock module
    returned by mock_import_swhmodule).
    
    This also comes with a change in expected test data structure for sql
    script; the `tests/data/<mod>` is now expected to store the sql script
    files in a dedicated `sql/` sub directory instead of having them
    directly in the `<mod>` directory so that the get_sql_for_package() can
    be used in tests unmocked.

commit 447027525f976aec9ef117a4502e6495da0628eb
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 1a187ca7787503d41ac13b2678b29f332d8fd200
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/339/ for more details.

Build is green

Patch application report for D7189 (id=26056)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..086932d
Fast-forward
 MANIFEST.in                                        |   2 +
 conftest.py                                        |   4 +-
 docs/db.rst                                        | 176 ++++++++++++
 docs/index.rst                                     |   1 +
 swh/core/cli/db.py                                 | 202 +++++++++++++-
 swh/core/db/db_utils.py                            | 300 +++++++++++++++++++--
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++-----
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  55 ++++
 .../tests/data/cli/{ => sql}/0-superuser-init.sql  |   0
 .../data/cli/{1-schema.sql => sql/30-schema.sql}   |   2 +-
 .../data/cli/{3-func.sql => sql/40-funcs.sql}      |   0
 .../tests/data/cli/{4-data.sql => sql/50-data.sql} |   2 +-
 .../db/tests/data/cli_new/sql/0-superuser-init.sql |   1 +
 swh/core/db/tests/data/cli_new/sql/30-schema.sql   |   6 +
 swh/core/db/tests/data/cli_new/sql/40-funcs.sql    |   6 +
 swh/core/db/tests/data/cli_new/sql/50-data.sql     |   2 +
 .../db/tests/data/cli_new/sql/upgrades/001.sql     |   5 +
 .../db/tests/data/cli_new/sql/upgrades/002.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/003.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/004.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/005.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/006.sql     |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 210 +++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 189 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 29 files changed, 1333 insertions(+), 152 deletions(-)
 create mode 100644 docs/db.rst
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%)
 rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 086932dc3bc399759063ac7bf732db628f120c99
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 15 14:14:45 2022 +0100

    Prevent root conftest.py from loading swh.core at module loading time
    
    move the import statement in swhmain() body.
    Having the swh.core module loaded with the execution of conftest.py can
    have weird side effects, especially on pytest's discovery mechanism in
    the context of tox, namely make pytest load test files from the source
    directory rather than from the installed package in the tox venv,
    potentially leading to ImportPathMismatchError.

commit 77498766aac687b73f71750f2f8eec96fc9d84c5
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 15:54:25 2022 +0100

    Add a documentation section for swh.core.db
    
    this is far from completion.

commit 106b22771617808e72c9813b94b6153a6cffd12f
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 12:37:07 2022 +0100

    Make get_sql_for_package() return pathlib.Path objects
    
    and replace mock_package_sql by a mock_import_swhmodule fixture in favor
    of using the stock get_sql_for_package() function (on the mock module
    returned by mock_import_swhmodule).
    
    This also comes with a change in expected test data structure for sql
    script; the `tests/data/<mod>` is now expected to store the sql script
    files in a dedicated `sql/` sub directory instead of having them
    directly in the `<mod>` directory so that the get_sql_for_package() can
    be used in tests unmocked.

commit 1c5c9f37155be12fe5589c5ba65bf894990e2103
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit c17d64f5884a148a3656c1b0993e3dcf30c4fb56
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    This feature is only working for "new style" swh.core.db based
    datastores.

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/347/ for more details.

Build is green

Patch application report for D7189 (id=26064)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..3f0124d
Fast-forward
 MANIFEST.in                                        |   2 +
 conftest.py                                        |   4 +-
 docs/db.rst                                        | 176 ++++++++++++
 docs/index.rst                                     |   1 +
 swh/core/cli/db.py                                 | 202 +++++++++++++-
 swh/core/db/db_utils.py                            | 300 +++++++++++++++++++--
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++-----
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  55 ++++
 .../tests/data/cli/{ => sql}/0-superuser-init.sql  |   0
 .../data/cli/{1-schema.sql => sql/30-schema.sql}   |   2 +-
 .../data/cli/{3-func.sql => sql/40-funcs.sql}      |   0
 .../tests/data/cli/{4-data.sql => sql/50-data.sql} |   2 +-
 .../db/tests/data/cli_new/sql/0-superuser-init.sql |   1 +
 swh/core/db/tests/data/cli_new/sql/30-schema.sql   |   6 +
 swh/core/db/tests/data/cli_new/sql/40-funcs.sql    |   6 +
 swh/core/db/tests/data/cli_new/sql/50-data.sql     |   2 +
 .../db/tests/data/cli_new/sql/upgrades/001.sql     |   5 +
 .../db/tests/data/cli_new/sql/upgrades/002.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/003.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/004.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/005.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/006.sql     |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 210 +++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 189 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 29 files changed, 1333 insertions(+), 152 deletions(-)
 create mode 100644 docs/db.rst
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%)
 rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 3f0124d67773f14f811c8eabc28eb3341e31dc41
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 15 14:14:45 2022 +0100

    Prevent root conftest.py from loading swh.core at module loading time
    
    move the import statement in swhmain() body.
    Having the swh.core module loaded with the execution of conftest.py can
    have weird side effects, especially on pytest's discovery mechanism in
    the context of tox, namely make pytest load test files from the source
    directory rather than from the installed package in the tox venv,
    potentially leading to ImportPathMismatchError.

commit 6a50f58a5e184db17ec73a72ce2e75b926b0c50a
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 15:54:25 2022 +0100

    Add a documentation section for swh.core.db
    
    this is far from completion.

commit 9a0c06981df822c9319e5e31cc32acac13425670
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 15:31:30 2022 +0100

    Replace mock_package_sql by a mock_import_swhmodule fixture
    
    in favor of using (and thus testing) the stock get_sql_for_package()
    function (on the mock module returned by mock_import_swhmodule).
    
    This comes with a change in expected test data structure for sql
    script; the `tests/data/<mod>` is now expected to store the sql script
    files in a dedicated `sql/` sub directory instead of having them
    directly in the `<mod>` directory so that the get_sql_for_package() can
    be used in tests unmocked.

commit b429a72c60984994b95e762f4bf3316ab466ced0
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 12:37:07 2022 +0100

    Make get_sql_for_package() return pathlib.Path objects

commit c11a4269ea967527c2e3424078288ab228b43eac
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 16e2fd4992efa439f074e11be4198cf5b0053aff
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    This feature is only working for "new style" swh.core.db based
    datastores.

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/352/ for more details.

Build is green

Patch application report for D7189 (id=26068)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..be2280a
Fast-forward
 MANIFEST.in                                        |   2 +
 conftest.py                                        |   4 +-
 docs/db.rst                                        | 176 ++++++++++++
 docs/index.rst                                     |   1 +
 swh/core/cli/db.py                                 | 202 +++++++++++++-
 swh/core/db/db_utils.py                            | 300 +++++++++++++++++++--
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++-----
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  55 ++++
 .../tests/data/cli/{ => sql}/0-superuser-init.sql  |   0
 .../data/cli/{1-schema.sql => sql/30-schema.sql}   |   2 +-
 .../data/cli/{3-func.sql => sql/40-funcs.sql}      |   0
 .../tests/data/cli/{4-data.sql => sql/50-data.sql} |   2 +-
 .../db/tests/data/cli_new/sql/0-superuser-init.sql |   1 +
 swh/core/db/tests/data/cli_new/sql/30-schema.sql   |   6 +
 swh/core/db/tests/data/cli_new/sql/40-funcs.sql    |   6 +
 swh/core/db/tests/data/cli_new/sql/50-data.sql     |   2 +
 .../db/tests/data/cli_new/sql/upgrades/001.sql     |   5 +
 .../db/tests/data/cli_new/sql/upgrades/002.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/003.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/004.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/005.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/006.sql     |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 210 +++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 189 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 29 files changed, 1333 insertions(+), 152 deletions(-)
 create mode 100644 docs/db.rst
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%)
 rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit be2280aca953fc8512acd920a870e5602a75c2f0
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 15 14:14:45 2022 +0100

    Prevent root conftest.py from loading swh.core at module loading time
    
    move the import statement in swhmain() body.
    Having the swh.core module loaded with the execution of conftest.py can
    have weird side effects, especially on pytest's discovery mechanism in
    the context of tox, namely make pytest load test files from the source
    directory rather than from the installed package in the tox venv,
    potentially leading to ImportPathMismatchError.

commit 6b61db4e4e58165393069290218860434cc13885
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 15:54:25 2022 +0100

    Add a documentation section for swh.core.db
    
    this is far from completion.

commit 6a88f4d5caed35e838cc240c381b3397e5a74f5a
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 15:31:30 2022 +0100

    Replace mock_package_sql by a mock_import_swhmodule fixture
    
    in favor of using (and thus testing) the stock get_sql_for_package()
    function (on the mock module returned by mock_import_swhmodule).
    
    This comes with a change in expected test data structure for sql
    script; the `tests/data/<mod>` is now expected to store the sql script
    files in a dedicated `sql/` sub directory instead of having them
    directly in the `<mod>` directory so that the get_sql_for_package() can
    be used in tests unmocked.

commit 06ea2290c44551ba1c6edcb07844aecd23b92b62
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 12:37:07 2022 +0100

    Make get_sql_for_package() return pathlib.Path objects

commit 893c93a958b239dddb6b42db64635b0e210892cd
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 16e2fd4992efa439f074e11be4198cf5b0053aff
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    This feature is only working for "new style" swh.core.db based
    datastores.

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/356/ for more details.

Build is green

Patch application report for D7189 (id=26087)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..9c9475f
Fast-forward
 MANIFEST.in                                        |   2 +
 conftest.py                                        |   4 +-
 docs/db.rst                                        | 176 ++++++++++++
 docs/index.rst                                     |   1 +
 requirements-db-pytestplugin.txt                   |   2 +-
 swh/core/cli/db.py                                 | 202 +++++++++++++-
 swh/core/db/db_utils.py                            | 300 +++++++++++++++++++--
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++-----
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  55 ++++
 .../tests/data/cli/{ => sql}/0-superuser-init.sql  |   0
 .../data/cli/{1-schema.sql => sql/30-schema.sql}   |   2 +-
 .../data/cli/{3-func.sql => sql/40-funcs.sql}      |   0
 .../tests/data/cli/{4-data.sql => sql/50-data.sql} |   2 +-
 .../db/tests/data/cli_new/sql/0-superuser-init.sql |   1 +
 swh/core/db/tests/data/cli_new/sql/30-schema.sql   |   6 +
 swh/core/db/tests/data/cli_new/sql/40-funcs.sql    |   6 +
 swh/core/db/tests/data/cli_new/sql/50-data.sql     |   2 +
 .../db/tests/data/cli_new/sql/upgrades/001.sql     |   5 +
 .../db/tests/data/cli_new/sql/upgrades/002.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/003.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/004.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/005.sql     |   4 +
 .../db/tests/data/cli_new/sql/upgrades/006.sql     |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 210 +++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 189 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 30 files changed, 1334 insertions(+), 153 deletions(-)
 create mode 100644 docs/db.rst
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{ => sql}/0-superuser-init.sql (100%)
 rename swh/core/db/tests/data/cli/{1-schema.sql => sql/30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => sql/40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => sql/50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/sql/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/sql/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 9c9475f769cf464ca9e74525a59ae1c91b107f25
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 15 14:14:45 2022 +0100

    Prevent root conftest.py from loading swh.core at module loading time
    
    move the import statement in swhmain() body.
    Having the swh.core module loaded with the execution of conftest.py can
    have weird side effects, especially on pytest's discovery mechanism in
    the context of tox, namely make pytest load test files from the source
    directory rather than from the installed package in the tox venv,
    potentially leading to ImportPathMismatchError.

commit 1bee4ce598c2768e807a8431a427008c8a690ff3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 15:54:25 2022 +0100

    Add a documentation section for swh.core.db
    
    this is far from completion.

commit 84b63d5b1da735b5ca4ad12023106f48bbcb07ee
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 15:31:30 2022 +0100

    Replace mock_package_sql by a mock_import_swhmodule fixture
    
    in favor of using (and thus testing) the stock get_sql_for_package()
    function (on the mock module returned by mock_import_swhmodule).
    
    This comes with a change in expected test data structure for sql
    script; the `tests/data/<mod>` is now expected to store the sql script
    files in a dedicated `sql/` sub directory instead of having them
    directly in the `<mod>` directory so that the get_sql_for_package() can
    be used in tests unmocked.

commit 25fee94847d3cb8de011cd34334eb4aac5628ce1
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Feb 16 12:37:07 2022 +0100

    Make get_sql_for_package() return pathlib.Path objects

commit 5822dab0ea427c11df68b5669bb60ac0d7e194f7
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 16e2fd4992efa439f074e11be4198cf5b0053aff
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    This feature is only working for "new style" swh.core.db based
    datastores.

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/360/ for more details.