Details

Reviewers

ardumont

Group Reviewers

Reviewers

Maniphest Tasks

T3894: Add a 'swh db upgrade' cli

Commits

rDCORE7e07b1f36326: Make 'swh db init' fill the dbversion table

Summary

for this feature to work properly, the backend datastore package must
provide the 'get_datastore' factory function in its root namespace and
the datastore instance must provide a 'get_current_version()' method.

This also means the canonical cli usage for the swh db init is now
using the config file (via --config-file or SWH_CONFIG_FILENAME)
instead of giving only the db cnx string (--db-name), so that the
backend datastore class can be instanciated using the get_datastore()
factory function.

However a '--initial-version' cli option has been added to 'swh db init'
to make it easier, especially for unit tests.

Also use a standard postgresql db fixture in test_cli
instead of a custom 'test_db' (which did not truncate dbversion and
origin tables, not sure what the intent was for this).

Depends on D7170
Related to T3894

Diff Detail

Repository

rDCORE Foundations and core functionalities

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Feb 2 2022, 1:52 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 2 2022, 1:52 PM

douardda added a child revision: D7064: Add a 'swh db upgrade' command.Feb 2 2022, 1:53 PM

Build has FAILED

Patch application report for D7063 (id=25613)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..a65510a
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 330 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  73 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 659 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit a65510a5e1cbc9aca48457bbb413c83906fa1e05
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 9f5b365b013c44b9219eddc4d4f92c06906c1d73
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 886c8c545fa424db720b4ec3cd1c8ec2de4e3952
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit cbc634d04ee9e57cdc42ebab2e087e1eac0031b5
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 4f41cc905d805953363688316a088bc84100e181
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit 6c665e38e6579a02839eef6a8a6b60dc332be51c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/268/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/268/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 2 2022, 1:54 PM

Harbormaster failed remote builds in B26443: Diff 25613!

rebase + fix

Build has FAILED

Patch application report for D7063 (id=25620)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..a4e3c6f
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  73 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 658 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit a4e3c6ff54903248fe2c581a2c920146208ad348
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 5ad23667d2f1acfef602236ebfe7e18ff72695d6
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 54c53dbdc2574e5f8d9f699505a3217708bf8b54
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/273/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/273/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 2 2022, 2:37 PM

Harbormaster failed remote builds in B26450: Diff 25620!

rebase

Build has FAILED

Patch application report for D7063 (id=25623)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..18aceb1
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  73 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 658 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 18aceb1ebdbd670d52efb0caf5d292a3fce6ba77
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/276/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/276/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 2 2022, 2:46 PM

Harbormaster failed remote builds in B26453: Diff 25623!

fix

Build has FAILED

Patch application report for D7063 (id=25625)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..23eecb8
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 657 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 23eecb8b0c95c57143abdeedc8fa7c12d5b715a8
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/278/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/278/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 2 2022, 2:53 PM

Harbormaster failed remote builds in B26455: Diff 25625!

Build has FAILED

Patch application report for D7063 (id=25625)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..23eecb8
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 11 files changed, 657 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 23eecb8b0c95c57143abdeedc8fa7c12d5b715a8
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/280/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/280/console

add forgotten data files

Build is green

Patch application report for D7063 (id=25628)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..471e411
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   5 +
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 15 files changed, 675 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 471e41184ef3e8f983fa55043d8c3daebbf87e8c
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/281/ for more details.

Harbormaster completed remote builds in B26458: Diff 25628.Feb 2 2022, 3:21 PM

douardda requested review of this revision.Feb 2 2022, 3:21 PM

small cleanup in cli_new/50-data.sql

Build is green

Patch application report for D7063 (id=25630)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..3fbb230
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 329 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 123 +++++---
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 15 files changed, 672 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 3fbb2305a22a6ef447b3634c09bf743eaba0a25e
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 981ae1d754edeea5cda2ff5e5eac957d7c376dfb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit f375b231aa73d934fb44c7d1071c20d4121a6186
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit c75374bafd77dfff82904985b935bbd035147cbc
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 5c5b981020c4b3e38f89f112480b7297cceded5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit d3b7b081fd3eda460d11d1d6d70f2b52fa3e0ef3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/283/ for more details.

Harbormaster completed remote builds in B26460: Diff 25630.Feb 2 2022, 3:25 PM

rebase

Build is green

Patch application report for D7063 (id=25635)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..1a4ab4e
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 264 ++++++++---------
 swh/core/db/db_utils.py                            | 326 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 123 ++++----
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 15 files changed, 669 insertions(+), 239 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 1a4ab4e856b2d5b7c97bc955c8d0373a95a0fd66
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 58e4f342eeeab1638e57d090d584435c91cf506d
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 2f2b047ffd906acf0d64d8f0d94e192e49759480
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit 7774e8e504afe02b46dc075e3ce9cc82bcc59ab8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 36bc18267c911abb85b6a1cfc3321942e0359369
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit de1c894e3fa88b3e1e2ba1e7c7dbfc77f7ea4b51
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/288/ for more details.

Harbormaster completed remote builds in B26465: Diff 25635.Feb 2 2022, 3:33 PM

rebase

Build is green

Patch application report for D7063 (id=25639)

Could not rebase; Attempt merge onto aba5c80765...

Updating aba5c80..a8a8214
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 289 +++++++++---------
 swh/core/db/db_utils.py                            | 326 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 123 ++++----
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 15 files changed, 684 insertions(+), 249 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit a8a821485532a09a4b2fecbb897d70aeb6d1cc1d
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit ba7318d74ef473a57c2791fad0ae32c55a74e6cf
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 7036f2d8cf38f8d5b5cb9a2df02235012e12ddcd
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit d1ab8a3ab5cfef97918b5168c0fa15f5c26c0e18
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit b11933f997e32db1b59be095a4dfe5d0c3be912b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit de1c894e3fa88b3e1e2ba1e7c7dbfc77f7ea4b51
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit e4117fcf51a256c5b708f909c818f452c794c0f8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/292/ for more details.

Harbormaster completed remote builds in B26469: Diff 25639.Feb 2 2022, 3:55 PM

vlorentz added a subscriber: vlorentz.Feb 2 2022, 3:58 PM

vlorentz added inline comments.

swh/core/cli/db.py
198–211	avoids accidentally catching unrelated exceptions
215–218
swh/core/db/db_utils.py
227–229	shouldn't it be reported to the caller function?
swh/core/db/tests/data/cli/30-schema.sql
2 ↗	(On Diff #25639)	why?
swh/core/db/tests/test_cli.py
225–227	toplevel
245–251	maybe?

olasd added a subscriber: olasd.Feb 2 2022, 5:16 PM

olasd added inline comments.

swh/core/db/tests/test_cli.py
245–251	or directly use `yaml.dumps({module_name: {"cls": "postgresql", "db": conninfo}})`

I'm not sure I like this change. What is the advantage of this over keeping an insert into dbversion ... statement in the SQL files, which would be used when the database is initialized?

rebase

Build is green

Patch application report for D7063 (id=25934)

Could not rebase; Attempt merge onto 5046a95f05...

Updating 5046a95..20e63a8
Fast-forward
 MANIFEST.in                                        |   3 +-
 swh/core/cli/db.py                                 | 289 +++++++++---------
 swh/core/db/db_utils.py                            | 326 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++
 swh/core/db/tests/conftest.py                      |  40 +++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 123 ++++----
 swh/core/db/tests/test_db_utils.py                 |  72 +++++
 swh/core/sql/log-schema.sql                        |  33 ---
 swh/core/tests/test_utils.py                       | 193 ++++++------
 swh/core/utils.py                                  |  16 +-
 17 files changed, 793 insertions(+), 349 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py
 delete mode 100644 swh/core/sql/log-schema.sql

Changes applied before test

commit 20e63a8c5105972d402264beb8d0fb84704142cd
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit ec245207af61185298b087955bc609a0704a126c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 93c9c701d28bf38b37bcf8ba5607fad3116c0087
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialise
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if avalable), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be bacward compatible with eisting swh datastore modules (i.e.
    not adapter to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

commit 3139d3344e232db83edacc95884604e47f0461cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 12:10:43 2022 +0100

    Fix a bug in numfile_sortkey and add tests for it

commit 68dc6811967ff0f6fb1296f9687d22d7f0fb684b
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Feb 11 11:59:56 2022 +0100

    Convert test_utils to pytest

commit 3e43730df6150168ca99cb083bd2e60feb4a2771
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:54:39 2022 +0100

    Make `swh db init` retrieve the db cnx uri from the config file

commit 459da96c1e6b6044deb874e0e668606c3da3e43c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 14:49:41 2022 +0100

    Move utility functions from cli.db to db.db_utils

commit 932eed4863baedf011b4a59d0227331d37db1dfd
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 10:07:35 2022 +0100

    Remove the old and unused log-schema.sql file

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/301/ for more details.

Harbormaster completed remote builds in B26811: Diff 25934.Feb 11 2022, 12:29 PM

douardda marked 4 inline comments as done.Feb 11 2022, 2:29 PM

douardda added inline comments.

swh/core/db/tests/data/cli/30-schema.sql
2 ↗	(On Diff #25639)	why? because I want now to catch cases where this table already exists while it should not (according the ongoing refactoring in handling this table).
swh/core/db/tests/test_cli.py
245–251	if we do not want to craft the yaml ourselves (which I find harmless in this case, but meh) , then olasd's solution is indeed better.

rebase

Build has FAILED

Patch application report for D7063 (id=25946)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..cfe1e60
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 116 ++++++++++++++-
 swh/core/db/db_utils.py                            | 164 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++++
 swh/core/db/tests/conftest.py                      |  40 +++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 116 +++++++++------
 swh/core/db/tests/test_db_utils.py                 |  72 +++++++++
 14 files changed, 497 insertions(+), 60 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py

Changes applied before test

commit cfe1e6059e85bbe8d1ad39a68f9953b478dcb294
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 2e184856ac6ff8af359d6112441ccbf837e6fed8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 943a813e318248e0c51a35d7866dbb145290c82a
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/305/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/305/console

Harbormaster failed remote builds in B26823: Diff 25946!Feb 11 2022, 2:35 PM

typos

Build is green

Patch application report for D7063 (id=25955)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..330a9d3
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 116 ++++++++++++++-
 swh/core/db/db_utils.py                            | 164 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++++
 swh/core/db/tests/conftest.py                      |  40 +++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 116 +++++++++------
 swh/core/db/tests/test_db_utils.py                 |  72 +++++++++
 14 files changed, 497 insertions(+), 60 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py

Changes applied before test

commit 330a9d31f5f188dbc457f6cda349bcafa09e396e
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit 2e184856ac6ff8af359d6112441ccbf837e6fed8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit 943a813e318248e0c51a35d7866dbb145290c82a
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/310/ for more details.

Harbormaster completed remote builds in B26831: Diff 25955.Feb 11 2022, 4:29 PM

ardumont added a subscriber: ardumont.Feb 14 2022, 1:39 PM

ardumont added inline comments.

swh/core/cli/db.py
215	/me likes destructuring (it's less resilient if you change the tuple within with more entries though ;)
swh/core/db/db_utils.py
218
237	How do we distinguish a successfull call from a failed one here? I suppose that joins back @vlorentz's question about reporting the failure to the caller. In any case, it should probably be mentioned in the docstring (what the caller should expect in the failure scenario).

instead of a custom 'test_db' (which did not truncate dbversion and
origin tables, not sure what the intent was for this).

probably keeping the existing test behavior?

ardumont added inline comments.Feb 14 2022, 1:43 PM

swh/core/cli/db.py
207	in the failure case, does `get_current_version` exist?

lgtm

As usual, couple of suggestions/questions (non blocking) inline.
(Adaptations if deemed worthy can always be added on top of those commits later on)

This revision is now accepted and ready to land.Feb 14 2022, 1:46 PM

rebas

Build has FAILED

Patch application report for D7063 (id=25984)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..544e184
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 116 ++++++++++++++-
 swh/core/db/db_utils.py                            | 164 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++++
 swh/core/db/tests/conftest.py                      |  40 +++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 116 +++++++++------
 swh/core/db/tests/test_db_utils.py                 |  73 +++++++++
 14 files changed, 498 insertions(+), 60 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py

Changes applied before test

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/316/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/316/console

douardda edited parent revisions, added: D7170: Use a standard postgresql db fixture in test_cli; removed: D7062: Add support for dbversion and dbmodule handling in swh db init.Feb 14 2022, 1:51 PM

Harbormaster failed remote builds in B26859: Diff 25984!Feb 14 2022, 1:51 PM

douardda edited the summary of this revision. (Show Details)Feb 14 2022, 1:51 PM

douardda marked an inline comment as done.Feb 14 2022, 1:54 PM

Build is green

Patch application report for D7063 (id=25984)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..544e184
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 116 ++++++++++++++-
 swh/core/db/db_utils.py                            | 164 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++++
 swh/core/db/tests/conftest.py                      |  40 +++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 116 +++++++++------
 swh/core/db/tests/test_db_utils.py                 |  73 +++++++++
 14 files changed, 498 insertions(+), 60 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py

Changes applied before test

commit 544e184045403c735c33e6029f8bb7fa1a4df5cb
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/323/ for more details.

Harbormaster completed remote builds in B26859: Diff 25984.Feb 16 2022, 10:34 AM

rebase

Build is green

Patch application report for D7063 (id=26051)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..7e07b1f
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 | 118 ++++++++++++++-
 swh/core/db/db_utils.py                            | 166 ++++++++++++++++++++-
 swh/core/db/sql/35-dbmetadata.sql                  |  28 ++++
 swh/core/db/tests/conftest.py                      |  45 ++++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   2 +-
 .../db/tests/data/cli_new/0-superuser-init.sql     |   1 +
 swh/core/db/tests/data/cli_new/30-schema.sql       |   6 +
 swh/core/db/tests/data/cli_new/40-funcs.sql        |   6 +
 swh/core/db/tests/data/cli_new/50-data.sql         |   2 +
 swh/core/db/tests/test_cli.py                      | 118 ++++++++-------
 swh/core/db/tests/test_db_utils.py                 |  78 ++++++++++
 14 files changed, 511 insertions(+), 63 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (73%)
 create mode 100644 swh/core/db/tests/data/cli_new/0-superuser-init.sql
 create mode 100644 swh/core/db/tests/data/cli_new/30-schema.sql
 create mode 100644 swh/core/db/tests/data/cli_new/40-funcs.sql
 create mode 100644 swh/core/db/tests/data/cli_new/50-data.sql
 create mode 100644 swh/core/db/tests/test_db_utils.py

Changes applied before test

commit 7e07b1f363269dc0bdcf3c1e7070ff0c0bacfdc6
Author: David Douard <david.douard@sdfa3.org>
Date:   Tue Feb 1 14:51:43 2022 +0100

    Make 'swh db init' fill the dbversion table
    
    for this feature to work properly, the backend datastore package must
    provide the 'get_datastore' factory function in its root namespace and
    the datastore instance must provide a 'get_current_version()' method.
    
    This also means the canonical cli usage for the `swh db init` is now
    using the config file (via `--config-file` or `SWH_CONFIG_FILENAME`)
    instead of giving only the db cnx string (`--db-name`), so that the
    backend datastore class can be instanciated using the `get_datastore()`
    factory function.
    
    However a '--initial-version' cli option has been added to 'swh db init'
    to make it easier, especially for unit tests.

commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/342/ for more details.

Harbormaster completed remote builds in B26920: Diff 26051.Feb 16 2022, 3:07 PM

Closed by commit rDCORE7e07b1f36326: Make 'swh db init' fill the dbversion table (authored by douardda). · Explain WhyFeb 17 2022, 1:55 PM

This revision was automatically updated to reflect the committed changes.

douardda added a commit: rDCORE7e07b1f36326: Make 'swh db init' fill the dbversion table.

Make 'swh db init' fill the dbversion tableClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D7063 (id=25613)

Changes applied before test

Patch application report for D7063 (id=25620)

Changes applied before test

Patch application report for D7063 (id=25623)

Changes applied before test

Patch application report for D7063 (id=25625)

Changes applied before test

Patch application report for D7063 (id=25625)

Changes applied before test

Patch application report for D7063 (id=25628)

Changes applied before test

Patch application report for D7063 (id=25630)

Changes applied before test

Patch application report for D7063 (id=25635)

Changes applied before test

Patch application report for D7063 (id=25639)

Changes applied before test

Patch application report for D7063 (id=25934)

Changes applied before test

Patch application report for D7063 (id=25946)

Changes applied before test

Patch application report for D7063 (id=25955)

Changes applied before test

Patch application report for D7063 (id=25984)

Changes applied before test

Patch application report for D7063 (id=25984)

Changes applied before test

Patch application report for D7063 (id=26051)

Changes applied before test

Revision ContentsChangeset List

Diff 26092

swh/core/cli/db.py

swh/core/db/db_utils.py

swh/core/db/tests/data/cli/50-data.sql

swh/core/db/tests/data/cli_new/0-superuser-init.sql

swh/core/db/tests/data/cli_new/30-schema.sql

swh/core/db/tests/data/cli_new/40-funcs.sql

swh/core/db/tests/data/cli_new/50-data.sql

swh/core/db/tests/test_cli.py

swh/core/db/tests/test_db_utils.py

Make 'swh db init' fill the dbversion table
ClosedPublic
Actions

Revision Contents
Changeset List