Page MenuHomeSoftware Heritage
Differential D7170

Use a standard postgresql db fixture in test_cli
ClosedPublic
Actions

Authored by douardda on Feb 14 2022, 1:49 PM.

Details

Reviewers
ardumont
Group Reviewers
Reviewers
Commits
rDCOREf8a07dfa0bf6: Use a standard postgresql db fixture in test_cli
Summary

instead of a custom 'test_db' (which did not truncate dbversion and
origin tables, not sure what the intent was for this).

Depends on D7062

Diff Detail

Event Timeline

douardda created this revision.Feb 14 2022, 1:49 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 14 2022, 1:49 PM
swh-public-ci added a comment.Feb 14 2022, 1:50 PM

Build has FAILED

Patch application report for D7170 (id=25983)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..adeb54f
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 |  67 +++++++++++
 swh/core/db/db_utils.py                            | 130 +++++++++++++++++++--
 swh/core/db/sql/35-dbmetadata.sql                  |  28 +++++
 swh/core/db/tests/conftest.py                      |  40 +++++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   0
 swh/core/db/tests/test_cli.py                      |  71 ++++-------
 swh/core/db/tests/test_db_utils.py                 |  78 +++++++++++++
 10 files changed, 363 insertions(+), 55 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (100%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

Link to build: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/315/
See console output for more information: https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/315/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 14 2022, 1:50 PM
Harbormaster failed remote builds in B26858: Diff 25983!
douardda added a child revision: D7063: Make 'swh db init' fill the dbversion table.Feb 14 2022, 1:51 PM
swh-public-ci added a comment.Feb 15 2022, 2:22 PM

Build is green

Patch application report for D7170 (id=25983)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..adeb54f
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 |  67 +++++++++++
 swh/core/db/db_utils.py                            | 130 +++++++++++++++++++--
 swh/core/db/sql/35-dbmetadata.sql                  |  28 +++++
 swh/core/db/tests/conftest.py                      |  40 +++++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   0
 swh/core/db/tests/test_cli.py                      |  71 ++++-------
 swh/core/db/tests/test_db_utils.py                 |  78 +++++++++++++
 10 files changed, 363 insertions(+), 55 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (100%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit adeb54f770209284f0924a7e4d5bd5977a567527
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit d50695c3feb57b1e21d93e78364a943232fe91ff
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/320/ for more details.

Harbormaster completed remote builds in B26858: Diff 25983.Feb 15 2022, 2:22 PM
douardda requested review of this revision.Feb 15 2022, 2:22 PM
ardumont accepted this revision.Feb 15 2022, 3:13 PM
This revision is now accepted and ready to land.Feb 15 2022, 3:13 PM
douardda updated this revision to Diff 26050.Feb 16 2022, 3:04 PM

rebase

swh-public-ci added a comment.Feb 16 2022, 3:06 PM

Build is green

Patch application report for D7170 (id=26050)

Could not rebase; Attempt merge onto 3139d3344e...

Updating 3139d33..f8a07df
Fast-forward
 MANIFEST.in                                        |   2 +
 swh/core/cli/db.py                                 |  69 ++++++++++-
 swh/core/db/db_utils.py                            | 132 +++++++++++++++++++--
 swh/core/db/sql/35-dbmetadata.sql                  |  28 +++++
 swh/core/db/tests/conftest.py                      |  45 +++++++
 .../tests/data/cli/{1-schema.sql => 30-schema.sql} |   2 +-
 .../db/tests/data/cli/{3-func.sql => 40-funcs.sql} |   0
 .../db/tests/data/cli/{4-data.sql => 50-data.sql}  |   0
 swh/core/db/tests/test_cli.py                      |  73 ++++--------
 swh/core/db/tests/test_db_utils.py                 |  83 +++++++++++++
 10 files changed, 376 insertions(+), 58 deletions(-)
 create mode 100644 swh/core/db/sql/35-dbmetadata.sql
 rename swh/core/db/tests/data/cli/{1-schema.sql => 30-schema.sql} (87%)
 rename swh/core/db/tests/data/cli/{3-func.sql => 40-funcs.sql} (100%)
 rename swh/core/db/tests/data/cli/{4-data.sql => 50-data.sql} (100%)
 create mode 100644 swh/core/db/tests/test_db_utils.py
Changes applied before test
commit f8a07dfa0bf668905d66ad88b25b2e1f3f569177
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 17:54:01 2022 +0100

    Use a standard postgresql db fixture in test_cli
    
    instead of a custom 'test_db' (which did not truncate dbversion and
    origin tables, not sure what the intent was for this).

commit c4bd270c3000df52176dafbf5c7f16e283e918c8
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 15:02:51 2022 +0100

    Add support for dbversion and dbmodule handling in swh db init
    
    for now, managing the dbversion table (storing the db schema upgrade
    history) is partially the responsibility of the swh modules implementing
    a datastore (swh-storage etc.), and the actual swh module was not stored
    in the database.
    
    This adds support for both the dbversion and dbmodule tables management
    in swh.core.db.
    
    The `swh db init` command now creates both these tables and initialize
    their values if possible.
    For this, it introduces a new common "API" for swh.core.db based datastores:
    
    - a swh module implementing a datastore is expected to have a
      `get_datastore` function in its top namespace; this function is
      typically the actual `get_storage`, `get_scheduler` etc. functions
    - this factory function is expected to use the "postgresql" cls for the
      datastore based on swh.core.db,
    - the datastore instance (eg. instance of the swh.storage.postgresql.Storage
      class) is expected to have a `get_current_version()` method (returning
      an int).
    
    This revision also provides a new `swh db version` command displaying
    the current (code) version (if available), the db version (possibly the whole
    version history) and flavor (if any) for the datastore given as argument.
    
    It should be backward compatible with existing swh datastore modules (i.e.
    not yet adapted to this new API).

commit b1e449fd91ebcc403966a79fd62d449921fcf550
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Jan 28 16:00:43 2022 +0100

    Refactor the mock_package_sql fixture to allow multiple sql script set
    
    will be used to implement several test scenarios in following commits.
    Also rename existing sql scripts with a "legit" numbering scheme.

See https://jenkins.softwareheritage.org/job/DCORE/job/tests-on-diff/341/ for more details.

Harbormaster completed remote builds in B26919: Diff 26050.Feb 16 2022, 3:06 PM
Closed by commit rDCOREf8a07dfa0bf6: Use a standard postgresql db fixture in test_cli (authored by douardda). · Explain WhyFeb 17 2022, 1:55 PM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDCOREf8a07dfa0bf6: Use a standard postgresql db fixture in test_cli.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
core/
db/
tests/
data/
cli/
2 lines--
37 lines99%
Loading...

Diff 25983

View Options

swh/core/db/tests/data/cli/30-schema.sql

Loading...
View Options

swh/core/db/tests/test_cli.py

Loading...
About · Developer information · Legal