Page MenuHomeSoftware Heritage

Refactor swh.core.db.pytest_plugin
ClosedPublic

Authored by douardda on Feb 11 2022, 12:47 PM.

Details

Summary

update the postgresql_fact fixture to be able to use new-ish
pytest_postgresql ability to recreate databases between tests from a
template database.

The ides is to stop using the fragile and confusing "reset" of the
database between tests, which means de facto keep a stateful database
between tests, which make tests result depend on de facto order of
execution of these tests.

For example, test_smoke_test_people_db_up_and_reset cannot be executed
alone, since it depends on the fact test_smoke_test_people_db_up has
been executed before so that the people's db has been reset as a result
of the test setup done by the fixture, which will happen if and only if
the database already exists (thus if another test using this db has been
executed before in this test session).

The old table truncate based behavior has been kept (using a no_db_drop
flag argument) in this diff for bw compatibility reasons, and also because
the new drop-and-create-from-template beahvior remains a bit slower,
which might be a problem for some test cases.

Tests for dependencies (swh-storage, etc.) should still pass with this
refactoring, but will be significantly slower.

A series of revisions for these dependencies is about to be proposed to
adapt them to the new db metadata (version, module, flavor) handling
feature coming with swh.core 2, as well as updating their tests to take
advantage of the template-based db creation in tests, which should bring
back test execution time to almost the same as before this refactoring.

This is needed to be able to implement and test support for the new 'swh
db upgrade' all this is actually about.

Depends on D7064

Event Timeline

Build is green

Patch application report for D7155 (id=25936)

Could not rebase; Attempt merge onto 5046a95f05...

Updating 5046a95..50c6933
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 334 ++++++++++--------
 swh/core/db/db_utils.py                            | 388 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 213 +++++++----
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  42 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  25 +-
 swh/core/db/tests/test_cli.py                      | 204 ++++++++---
 swh/core/db/tests/test_db_utils.py                 | 117 +++++++
 swh/core/sql/log-schema.sql                        |  33 --
 swh/core/tests/test_utils.py                       | 198 +++++------
 swh/core/utils.py                                  |  21 +-
 25 files changed, 1240 insertions(+), 413 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql
Changes applied before test
commit 50c6933f1fece490a2f9cc9380da0ea3621b2b2c
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 95f137c38bb41d6b60669402a82ac829145c5121
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 20e63a8c5105972d402264beb8d0fb84704142cd
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit ec245207af61185298b087955bc609a0704a126c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 93c9c701d28bf38b37bcf8ba5607fad3116c0087
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 3139d3344e232db83edacc95884604e47f0461cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 12:10:43 2022 +0100

    Fix a bug in numfile_sortkey and add tests for it

commit 68dc6811967ff0f6fb1296f9687d22d7f0fb684b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 11:59:56 2022 +0100

    Convert test_utils to pytest

commit 3e43730df6150168ca99cb083bd2e60feb4a2771
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit 459da96c1e6b6044deb874e0e668606c3da3e43c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit 932eed4863baedf011b4a59d0227331d37db1dfd
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/303/ for more details.

rebase and print/pass statements

Build has FAILED

Patch application report for D7155 (id=25949)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..1b1df67
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 179 +++++++++++++++-
 swh/core/db/db_utils.py                            | 228 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 213 +++++++++++++------
 swh/core/db/sql/35-dbmetadata.sql                  |  28 +++
 swh/core/db/tests/conftest.py                      |  42 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  25 ++-
 swh/core/db/tests/test_cli.py                      | 197 +++++++++++++-----
 swh/core/db/tests/test_db_utils.py                 | 117 +++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 24 files changed, 954 insertions(+), 134 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 1b1df671cabe276c8a2c9116c791cf91edede8f9
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 5712a5d290966b13242d3a790bd29e6a442fa87b
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit cfe1e6059e85bbe8d1ad39a68f9953b478dcb294
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 2e184856ac6ff8af359d6112441ccbf837e6fed8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 943a813e318248e0c51a35d7866dbb145290c82a
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/307/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/307/console

Build has FAILED

Patch application report for D7155 (id=25952)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..e71baa9
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 179 +++++++++++++++-
 swh/core/db/db_utils.py                            | 228 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 213 +++++++++++++------
 swh/core/db/sql/35-dbmetadata.sql                  |  28 +++
 swh/core/db/tests/conftest.py                      |  42 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 ++-
 swh/core/db/tests/test_cli.py                      | 197 +++++++++++++-----
 swh/core/db/tests/test_db_utils.py                 | 117 +++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 24 files changed, 953 insertions(+), 136 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit e71baa97f684c75f530456fcddef6755e5de9bde
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 5712a5d290966b13242d3a790bd29e6a442fa87b
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit cfe1e6059e85bbe8d1ad39a68f9953b478dcb294
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 2e184856ac6ff8af359d6112441ccbf837e6fed8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 943a813e318248e0c51a35d7866dbb145290c82a
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/308/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/308/console

Build is green

Patch application report for D7155 (id=25957)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..2981fa0
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 179 +++++++++++++++-
 swh/core/db/db_utils.py                            | 228 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 213 +++++++++++++------
 swh/core/db/sql/35-dbmetadata.sql                  |  28 +++
 swh/core/db/tests/conftest.py                      |  42 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 ++-
 swh/core/db/tests/test_cli.py                      | 197 +++++++++++++-----
 swh/core/db/tests/test_db_utils.py                 | 117 +++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 24 files changed, 953 insertions(+), 136 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 2981fa08f812d020485603ca4303a17c53a718d0
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit fc451f2b86eff00fb5dea99c919d5187e5e3c5b5
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 330a9d31f5f188dbc457f6cda349bcafa09e396e
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 2e184856ac6ff8af359d6112441ccbf837e6fed8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 943a813e318248e0c51a35d7866dbb145290c82a
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/312/ for more details.

Build has FAILED

Patch application report for D7155 (id=25986)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..ddce7a1
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 179 +++++++++++++++-
 swh/core/db/db_utils.py                            | 228 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 213 +++++++++++++------
 swh/core/db/sql/35-dbmetadata.sql                  |  28 +++
 swh/core/db/tests/conftest.py                      |  42 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 ++-
 swh/core/db/tests/test_cli.py                      | 197 +++++++++++++-----
 swh/core/db/tests/test_db_utils.py                 | 118 +++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 24 files changed, 954 insertions(+), 136 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit ddce7a1485f39277da9773c4aed494459ffd12eb
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 528371a0f84aff02604ea51c6e24fe97ebd3de7a
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/318/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/318/console

ardumont added a subscriber: ardumont.

lgtm

couple of questions/remarks inline.

swh/core/db/pytest_plugin.py
70
128

or something ;)

177–190

to match the no_truncate_table no_<verb>_<entity>`?

187

missing some new params here.

257

docstring and type would be welcome ;)

(type: tentatively added)

261

what does that do? Is that a means to determine whether that's a folder or something (it looks that way from afar)?

In [2]: glob.has_magic?
Signature: glob.has_magic(s)
Docstring: <no docstring>
File:      /usr/lib/python3.9/glob.py

did not help ^

This revision is now accepted and ready to land.Feb 14 2022, 4:28 PM

Build is green

Patch application report for D7155 (id=26034)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..d187e7e
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 200 +++++++++++++++-
 swh/core/db/db_utils.py                            | 266 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 213 +++++++++++------
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  42 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 220 +++++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 166 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 25 files changed, 1089 insertions(+), 136 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit d187e7e4574bb190a3af8254ec116a89a59e34bf
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit dfa01610264a925ba02e33c6f2401d34fd34ec18
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/325/ for more details.

Build is green

Patch application report for D7155 (id=26041)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..9e599f0
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 200 ++++++++++++++-
 swh/core/db/db_utils.py                            | 271 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 213 ++++++++++------
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  42 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 220 +++++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 166 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 25 files changed, 1094 insertions(+), 136 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 9e599f03076f85b44ac0eb5d001ff6367a930276
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 1a187ca7787503d41ac13b2678b29f332d8fd200
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/332/ for more details.

douardda added inline comments.
swh/core/db/pytest_plugin.py
177–190

not fond of no_drop_db, even if it kind of look more consistent with no_truncate_tables.

douardda marked an inline comment as done.

typos and fixes suggested by ardumon

Build is green

Patch application report for D7155 (id=26045)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..4470275
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 200 ++++++++++++++-
 swh/core/db/db_utils.py                            | 271 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++------
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  42 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 220 +++++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 166 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 25 files changed, 1110 insertions(+), 137 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 447027525f976aec9ef117a4502e6495da0628eb
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 1a187ca7787503d41ac13b2678b29f332d8fd200
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    Thie feature is only working for "new style" swh.core.db based
    datastores.

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/336/ for more details.

swh/core/db/pytest_plugin.py
128

that one is on me!

177–190

(i'm not fond of any of those really ¯\_(ツ)_/¯)

195

Build is green

Patch application report for D7155 (id=26053)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..1c5c9f3
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 202 ++++++++++++++-
 swh/core/db/db_utils.py                            | 273 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++------
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  47 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 216 ++++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 174 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 25 files changed, 1120 insertions(+), 140 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 1c5c9f37155be12fe5589c5ba65bf894990e2103
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit c17d64f5884a148a3656c1b0993e3dcf30c4fb56
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    This feature is only working for "new style" swh.core.db based
    datastores.

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/344/ for more details.

Build is green

Patch application report for D7155 (id=26061)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..c11a426
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 202 ++++++++++++++-
 swh/core/db/db_utils.py                            | 273 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++------
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  47 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 216 ++++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 174 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 25 files changed, 1120 insertions(+), 140 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit c11a4269ea967527c2e3424078288ab228b43eac
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 16e2fd4992efa439f074e11be4198cf5b0053aff
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    This feature is only working for "new style" swh.core.db based
    datastores.

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/349/ for more details.

douardda added inline comments.
swh/core/db/pytest_plugin.py
128

that one is on me!

:-)

douardda marked an inline comment as done.

typos

Build is green

Patch application report for D7155 (id=26065)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..893c93a
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 202 ++++++++++++++-
 swh/core/db/db_utils.py                            | 273 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++------
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  47 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 216 ++++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 174 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 25 files changed, 1120 insertions(+), 140 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 893c93a958b239dddb6b42db64635b0e210892cd
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 16e2fd4992efa439f074e11be4198cf5b0053aff
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    This feature is only working for "new style" swh.core.db based
    datastores.

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/353/ for more details.

update dependency on pytest_postgresql

Build is green

Patch application report for D7155 (id=26084)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..5822dab
Fast-forward
 MANIFEST.in                                        |   2 +
 requirements-db-pytestplugin.txt                   |   2 +-
 swh/core/cli/db.py                                 | 202 ++++++++++++++-
 swh/core/db/db_utils.py                            | 273 ++++++++++++++++++++-
 swh/core/db/pytest_plugin.py                       | 230 +++++++++++------
 swh/core/db/sql/35-dbversion.sql                   |  18 ++
 swh/core/db/sql/36-dbmodule.sql                    |  15 ++
 swh/core/db/tests/conftest.py                      |  47 ++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/data/cli_new/upgrades/001.sql    |   5 +
 swh/core/db/tests/data/cli_new/upgrades/002.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/003.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/004.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/005.sql    |   4 +
 swh/core/db/tests/data/cli_new/upgrades/006.sql    |   7 +
 .../db/tests/pytest_plugin/test_pytest_plugin.py   |  26 +-
 swh/core/db/tests/test_cli.py                      | 216 ++++++++++++----
 swh/core/db/tests/test_db_utils.py                 | 174 +++++++++++++
 swh/core/tests/test_utils.py                       |   5 +
 swh/core/utils.py                                  |   5 +
 26 files changed, 1121 insertions(+), 141 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbversion.sql
 create mode 100644 swh/core/db/sql/36-dbmodule.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/001.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/002.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/003.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/004.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/005.sql
 create mode 100644 swh/core/db/tests/data/cli_new/upgrades/006.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit 5822dab0ea427c11df68b5669bb60ac0d7e194f7
Author: David Douard <david.douard@sdfa3.org>
Date:   Mon Feb 7 15:46:48 2022 +0100

    Refactor swh.core.db.pytest_plugin
    
    update the postgresql_fact fixture to be able to use new-ish
    pytest_postgresql ability to recreate databases between tests from a
    template database.
    
    The ides is to stop using the fragile and confusing "reset" of the
    database between tests, which means de facto keep a stateful database
    between tests, which make tests result depend on de facto order of
    execution of these tests.
    
    For example, test_smoke_test_people_db_up_and_reset cannot be executed
    alone, since it depends on the fact test_smoke_test_people_db_up has
    been executed before so that the people's db has been reset as a result
    of the test setup done by the fixture, which will happen if and only if
    the database already exists (thus if another test using this db has been
    executed before in this test session).
    
    The old table truncate based behavior has been kept (using a no_db_drop
    flag argument) in this diff for bw compatibility reasons, and also because
    the new drop-and-create-from-template beahvior remains a bit slower,
    which might be a problem for some test cases.
    
    Tests for dependencies (swh-storage, etc.) should still pass with this
    refactoring, but will be significantly slower.
    
    A series of revisions for these dependencies is about to be proposed to
    adapt them to the new db metadata (version, module, flavor) handling
    feature coming with swh.core 2, as well as updating their tests to take
    advantage of the template-based db creation in tests, which should bring
    back test execution time to almost the same as before this refactoring.
    
    This is needed to be able to implement and test support for the new 'swh
    db upgrade' all this is actually about.

commit 16e2fd4992efa439f074e11be4198cf5b0053aff
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 15:43:09 2022 +0100

    Add a 'swh db upgrade' command
    
    this new command allows to handle the database migration from the
    current version to the latest version advertized but the backend
    datastore.
    
    This feature is only working for "new style" swh.core.db based
    datastores.

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/357/ for more details.

This revision was automatically updated to reflect the committed changes.