Page MenuHomeSoftware Heritage
Differential D5014

[RFC] Add basic migration tests for postgresql
Needs ReviewPublic
Actions

Authored by vlorentz on Feb 4 2021, 2:39 PM.

Details

Reviewers
None
Group Reviewers
Reviewers
Maniphest Tasks
T2076: Add tests for SQL migrations
Summary

This adds two test files:

  • test_postgresql_migrated.py applies an old schema definition, runs the migrations, then runs all the usual tests
  • test_postgresql_migration.py applies an old schema definition, inserts data, runs the migrations, and checks the data is still available

test_postgresql_migration.py will probably break in some releases as
it uses the old SQL with the new Python to insert, but it should be good
enough, and we can disable it in some releases when needed.

What do you think?

Test Plan

tests fail because they don't work in tox (I'll look into later if we want to pursue this way of testing migrations)

Diff Detail

Repository
rDSTO Storage manager
Branch
migration-tests
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 19700
Build 30575: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 30574: arc lint + arc unit

Event Timeline

vlorentz created this revision.Feb 4 2021, 2:39 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 4 2021, 2:39 PM
swh-public-ci added a comment.Feb 4 2021, 2:40 PM

Build has FAILED

Patch application report for D5014 (id=17885)

Could not rebase; Attempt merge onto 9a9f234e0a...

Updating 9a9f234e..27440f98
Fast-forward
 swh/storage/cassandra/common.py                    |   5 -
 swh/storage/cassandra/converters.py                |   2 +-
 swh/storage/cassandra/cql.py                       |   3 +-
 swh/storage/cassandra/storage.py                   |   3 +-
 swh/storage/postgresql/db.py                       |   9 +-
 .../tests/data/sql-v0.18.0/10-superuser-init.sql   |  27 +
 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql   |  22 +
 swh/storage/tests/data/sql-v0.18.0/20-enums.sql    |  23 +
 swh/storage/tests/data/sql-v0.18.0/30-schema.sql   | 499 +++++++++++
 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql    | 960 +++++++++++++++++++++
 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql  | 283 ++++++
 .../logical_replication/replication_source.sql     |  25 +
 swh/storage/tests/storage_tests.py                 |  59 +-
 swh/storage/tests/test_postgresql_migrated.py      |  63 ++
 swh/storage/tests/test_postgresql_migration.py     | 194 +++++
 swh/storage/utils.py                               |   5 +
 16 files changed, 2158 insertions(+), 24 deletions(-)
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/10-superuser-init.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/20-enums.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/30-schema.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/logical_replication/replication_source.sql
 create mode 100644 swh/storage/tests/test_postgresql_migrated.py
 create mode 100644 swh/storage/tests/test_postgresql_migration.py
Changes applied before test
commit 27440f9862b7df4e36b400a4fb74f1679d9ec6f4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:36:50 2021 +0100

    Add basic migration tests for postgresql
    
    This adds two test files:
    
    * `test_postgresql_migrated.py` applies an old schema definition, runs the
      migrations, then runs all the usual tests
    * `test_postgresql_migration.py` applies an old schema definition, inserts
      data, runs the migrations, and checks the data is still available
    
    `test_postgresql_migration.py` will probably break in some releases as
    it uses the old SQL with the new Python to insert, but it should be good
    enough, and we can disable it in some releases when needed.

commit e9441fef13c11c3eb500403275ba7337ed0c77e1
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:29:38 2021 +0100

    postgresql: Fix dbversion() to return the max version instead of a random one.

commit 508399ce2abf21f813acc9c56422cbbccca0ae3d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 13:59:09 2021 +0100

    storage_tests: recompute ids when evolving RawExtrinsicMetadata objects.
    
    For now this does nothing as RawExtrinsicMetadata has no 'id' field,
    but the equality assertions will become errors when the next version
    of swh.model is released.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1122/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1122/console

Harbormaster returned this revision to the author for changes because remote builds failed.Feb 4 2021, 2:40 PM
Harbormaster failed remote builds in B19004: Diff 17885!
vlorentz requested review of this revision.Feb 4 2021, 2:42 PM
vlorentz mentioned this in D5016: pytest_plugins: Make postgresql_fact accept List[str] in addition to str..Feb 4 2021, 2:44 PM
vlorentz edited the test plan for this revision. (Show Details)Feb 4 2021, 2:44 PM
vlorentz retitled this revision from Add basic migration tests for postgresql to [RFC] Add basic migration tests for postgresql.Feb 4 2021, 3:35 PM
vlorentz updated this revision to Diff 17970.Feb 8 2021, 2:19 PM

rebase

swh-public-ci added a comment.Feb 8 2021, 2:20 PM

Build has FAILED

Patch application report for D5014 (id=17970)

Could not rebase; Attempt merge onto b0383833fe...

Updating b0383833..ebe4dab0
Fast-forward
 swh/storage/cassandra/common.py                    |   5 -
 swh/storage/cassandra/converters.py                |   2 +-
 swh/storage/cassandra/cql.py                       |   3 +-
 swh/storage/cassandra/storage.py                   |   3 +-
 .../tests/data/sql-v0.18.0/10-superuser-init.sql   |  27 +
 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql   |  22 +
 swh/storage/tests/data/sql-v0.18.0/20-enums.sql    |  23 +
 swh/storage/tests/data/sql-v0.18.0/30-schema.sql   | 499 +++++++++++
 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql    | 960 +++++++++++++++++++++
 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql  | 283 ++++++
 .../logical_replication/replication_source.sql     |  25 +
 swh/storage/tests/storage_tests.py                 |  59 +-
 swh/storage/tests/test_postgresql_migrated.py      |  63 ++
 swh/storage/tests/test_postgresql_migration.py     | 194 +++++
 swh/storage/utils.py                               |   5 +
 15 files changed, 2150 insertions(+), 23 deletions(-)
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/10-superuser-init.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/20-enums.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/30-schema.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/logical_replication/replication_source.sql
 create mode 100644 swh/storage/tests/test_postgresql_migrated.py
 create mode 100644 swh/storage/tests/test_postgresql_migration.py
Changes applied before test
commit ebe4dab07feed7eb8ae03cd407980ba7bea78b37
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:36:50 2021 +0100

    Add basic migration tests for postgresql
    
    This adds two test files:
    
    * `test_postgresql_migrated.py` applies an old schema definition, runs the
      migrations, then runs all the usual tests
    * `test_postgresql_migration.py` applies an old schema definition, inserts
      data, runs the migrations, and checks the data is still available
    
    `test_postgresql_migration.py` will probably break in some releases as
    it uses the old SQL with the new Python to insert, but it should be good
    enough, and we can disable it in some releases when needed.

commit 75939e4f9b02e130224a90929eb3ed2ba0730592
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 13:59:09 2021 +0100

    storage_tests: recompute ids when evolving RawExtrinsicMetadata objects.
    
    For now this does nothing as RawExtrinsicMetadata has no 'id' field,
    but the equality assertions will become errors when the next version
    of swh.model is released.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1143/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1143/console

Harbormaster failed remote builds in B19091: Diff 17970!Feb 8 2021, 2:20 PM
vlorentz added a task: T2076: Add tests for SQL migrations.Feb 8 2021, 2:25 PM
vlorentz updated this revision to Diff 18605.Mar 4 2021, 4:05 PM

rebase

swh-public-ci added a comment.Mar 4 2021, 4:11 PM

Build has FAILED

Patch application report for D5014 (id=18605)

Rebasing onto 88ff2c2fa0...

Current branch diff-target is up to date.
Changes applied before test
commit e5fc49846c1d9ca9961b0b7edf7e0a993401daf6
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:36:50 2021 +0100

    Add basic migration tests for postgresql
    
    This adds two test files:
    
    * `test_postgresql_migrated.py` applies an old schema definition, runs the
      migrations, then runs all the usual tests
    * `test_postgresql_migration.py` applies an old schema definition, inserts
      data, runs the migrations, and checks the data is still available
    
    `test_postgresql_migration.py` will probably break in some releases as
    it uses the old SQL with the new Python to insert, but it should be good
    enough, and we can disable it in some releases when needed.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1177/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1177/console

Harbormaster failed remote builds in B19697: Diff 18605!Mar 4 2021, 4:11 PM
vlorentz updated this revision to Diff 18608.Mar 4 2021, 4:23 PM

rebase

vlorentz edited the test plan for this revision. (Show Details)Mar 4 2021, 4:23 PM
swh-public-ci added a comment.Mar 4 2021, 4:24 PM

Build has FAILED

Patch application report for D5014 (id=18608)

Could not rebase; Attempt merge onto 88ff2c2fa0...

Updating 88ff2c2f..e583c3cb
Fast-forward
 sql/upgrades/169.sql                               |  26 +
 swh/storage/postgresql/db.py                       |   5 +-
 swh/storage/postgresql/storage.py                  |   1 +
 swh/storage/sql/30-schema.sql                      |   4 +-
 swh/storage/sql/60-indexes.sql                     |   4 +
 .../tests/data/sql-v0.18.0/10-superuser-init.sql   |  27 +
 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql   |  22 +
 swh/storage/tests/data/sql-v0.18.0/20-enums.sql    |  23 +
 swh/storage/tests/data/sql-v0.18.0/30-schema.sql   | 499 +++++++++++
 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql    | 960 +++++++++++++++++++++
 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql  | 283 ++++++
 .../logical_replication/replication_source.sql     |  25 +
 swh/storage/tests/test_postgresql_migrated.py      |  63 ++
 swh/storage/tests/test_postgresql_migration.py     | 194 +++++
 14 files changed, 2134 insertions(+), 2 deletions(-)
 create mode 100644 sql/upgrades/169.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/10-superuser-init.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/15-flavor.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/20-enums.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/30-schema.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/40-funcs.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/60-indexes.sql
 create mode 100644 swh/storage/tests/data/sql-v0.18.0/logical_replication/replication_source.sql
 create mode 100644 swh/storage/tests/test_postgresql_migrated.py
 create mode 100644 swh/storage/tests/test_postgresql_migration.py
Changes applied before test
commit e583c3cb0efb0805af0b88547bc67b7841e00e2b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Feb 4 14:36:50 2021 +0100

    Add basic migration tests for postgresql
    
    This adds two test files:
    
    * `test_postgresql_migrated.py` applies an old schema definition, runs the
      migrations, then runs all the usual tests
    * `test_postgresql_migration.py` applies an old schema definition, inserts
      data, runs the migrations, and checks the data is still available
    
    `test_postgresql_migration.py` will probably break in some releases as
    it uses the old SQL with the new Python to insert, but it should be good
    enough, and we can disable it in some releases when needed.

commit 7ed181fe084152ebb2ad5b1792a8d2d9e5a1c429
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Feb 5 13:56:15 2021 +0100

    Add raw_extrinsic_metadata.id column in postgresql.
    
    For now, this has absolutely no effect on the API users,
    as rows are already deduplicated based on a subset of the
    fields hashed by the id.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1180/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1180/console

Harbormaster failed remote builds in B19700: Diff 18608!Mar 4 2021, 4:24 PM

Revision Contents
Changeset List

PathSize
swh/
storage/
tests/
data/
sql-v0.18.0/
27 lines
22 lines
23 lines
499 lines
960 lines
283 lines
logical_replication/
25 lines
63 lines
194 lines

Diff 18608

View Options

swh/storage/tests/data/sql-v0.18.0/10-superuser-init.sql

Loading...
View Options

swh/storage/tests/data/sql-v0.18.0/15-flavor.sql

Loading...
View Options

swh/storage/tests/data/sql-v0.18.0/20-enums.sql

Loading...
View Options

swh/storage/tests/data/sql-v0.18.0/30-schema.sql

Loading...
View Options

swh/storage/tests/data/sql-v0.18.0/40-funcs.sql

Loading...
View Options

swh/storage/tests/data/sql-v0.18.0/60-indexes.sql

Loading...
View Options

swh/storage/tests/data/sql-v0.18.0/logical_replication/replication_source.sql

Loading...
View Options

swh/storage/tests/test_postgresql_migrated.py

Loading...
View Options

swh/storage/tests/test_postgresql_migration.py

Loading...
About · Developer information · Legal