Page MenuHomeSoftware Heritage
Differential D3808

Move postgresql-related files to swh/storage/postgresql/
ClosedPublic
Actions

Authored by vlorentz on Aug 19 2020, 4:15 PM.

Details

Reviewers
anlambert
Group Reviewers
Reviewers
Commits
rDSTO40739076be0d: Move postgresql-related files to swh/storage/postgresql/
Summary

The asymmetry of having them directly in swh/storage/ while the cassandra backend
has them in swh/storage/cassandra/ has been bugging me for a while, so it's
time we finally do it.

Diff Detail

Repository
rDSTO Storage manager
Branch
move-pg
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 14682
Build 22622: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 22621: arc lint + arc unit

Event Timeline

vlorentz created this revision.Aug 19 2020, 4:15 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptAug 19 2020, 4:15 PM
swh-public-ci added a comment.Aug 19 2020, 4:20 PM

Build is green

Patch application report for D3808 (id=13415)

Could not rebase; Attempt merge onto bd925475d9...

Updating bd925475..3aff2dfd
Fast-forward
 sql/upgrades/160.sql                               |   41 +
 sql/upgrades/161.sql                               |   17 +
 swh/storage/__init__.py                            |    2 +-
 swh/storage/algos/diff.py                          |    6 +-
 swh/storage/algos/dir_iterators.py                 |    4 +-
 swh/storage/algos/revisions_walker.py              |   22 +-
 swh/storage/algos/snapshot.py                      |    2 +-
 swh/storage/backfill.py                            |    3 +-
 swh/storage/common.py                              |    9 +-
 swh/storage/in_memory.py                           |    2 +-
 swh/storage/postgresql/__init__.py                 |    0
 swh/storage/{ => postgresql}/converters.py         |   24 +-
 swh/storage/{ => postgresql}/db.py                 |    0
 swh/storage/{ => postgresql}/storage.py            |   18 +-
 swh/storage/sql/30-swh-schema.sql                  |    4 +-
 swh/storage/sql/60-swh-indexes.sql                 |   22 +
 swh/storage/tests/algos/test_revisions_walker.py   |    2 +-
 swh/storage/tests/storage_tests.py                 | 4123 ++++++++++++++++++++
 swh/storage/tests/test_api_client.py               |    2 +-
 swh/storage/tests/test_in_memory.py                |    4 +-
 swh/storage/tests/test_init.py                     |    6 +-
 swh/storage/tests/test_postgresql.py               |  256 ++
 ...converters.py => test_postgresql_converters.py} |    2 +-
 swh/storage/tests/test_revision_bw_compat.py       |    2 +-
 swh/storage/tests/test_storage.py                  |  250 +-
 25 files changed, 4525 insertions(+), 298 deletions(-)
 create mode 100644 sql/upgrades/160.sql
 create mode 100644 sql/upgrades/161.sql
 create mode 100644 swh/storage/postgresql/__init__.py
 rename swh/storage/{ => postgresql}/converters.py (94%)
 rename swh/storage/{ => postgresql}/db.py (100%)
 rename swh/storage/{ => postgresql}/storage.py (99%)
 create mode 100644 swh/storage/tests/storage_tests.py
 create mode 100644 swh/storage/tests/test_postgresql.py
 rename swh/storage/tests/{test_converters.py => test_postgresql_converters.py} (99%)
Changes applied before test
commit 3aff2dfd0a973dafd64067356b932868bfc4117a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 16:13:48 2020 +0200

    Move postgresql-related files to swh/storage/postgresql/

commit 6a53cb3ef3ad769ab015fa011e0a14702d1a14cf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 14:25:22 2020 +0200

    pg: Check revision.extra_headers is not null.
    
    Fixes a regression in 038a219f84d6b8a4f02b48f9ad3c5d823d097790,
    as it made the converter expect a list.
    
    When adding this column, we made it default to null instead of defaulting
    to an empty array, so existing records were initialized will null.
    
    This commit migrates these nulls to empty arrays, then adds a constraint
    to enforce it in the future.

commit ca3ee92da47be37c25439a49ef18c4a1d63edd58
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 14:23:54 2020 +0200

    converters: convert extra_headers to an empty list if it is None.

commit e2b14948da143925b0421415580df63e48d18a5b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 12:40:01 2020 +0200

    pg: Make date_neg_utc_offset is not null if date is not null.
    
    Fixes a regression in 038a219f84d6b8a4f02b48f9ad3c5d823d097790,
    as it made converters expect a boolean.
    
    We stopped writing this kind of nulls since we started using model
    objects for insertions, but didn't migrate existing data.
    
    This commit migrates these nulls to false, then adds a constraint
    to enforce it in the future.

commit 7dcd5708b0cffdacd0082288262ccc4d78cec2de
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 14:06:21 2020 +0200

    converters: convert neg_utc_offset to False if it is None.

commit e2f066549123f069bf0ae34d593038ffd834d9c4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 14:54:45 2020 +0200

    backfiller: Add missing 'extra_header' field.
    
    This field wasn't backfilled; and it wasn't caught by tests, because
    revisions in swh.journal.tests.journal_data.TEST_OBJECTS were had
    an empty extra_header field.
    
    Starting with swh-journal v0.4.3, one of the revisions in TEST_OBJECTS
    has a non-empty extra_header field, so it will become a test failure
    without this commit

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/828/ for more details.

Harbormaster completed remote builds in B14679: Diff 13415.Aug 19 2020, 4:20 PM
anlambert accepted this revision.Aug 19 2020, 4:47 PM
anlambert added a subscriber: anlambert.

+1, I agree it is time to reorganize source code this way.

This revision is now accepted and ready to land.Aug 19 2020, 4:47 PM
vlorentz updated this revision to Diff 13418.Aug 19 2020, 5:18 PM

remove duplicate test_storage.py

swh-public-ci added a comment.Aug 19 2020, 5:18 PM

Build has FAILED

Patch application report for D3808 (id=13418)

Could not rebase; Attempt merge onto bd925475d9...

Updating bd925475..40739076
Fast-forward
 sql/upgrades/160.sql                               |  41 ++++
 sql/upgrades/161.sql                               |  17 ++
 swh/storage/__init__.py                            |   2 +-
 swh/storage/algos/diff.py                          |   6 +-
 swh/storage/algos/dir_iterators.py                 |   4 +-
 swh/storage/algos/revisions_walker.py              |  22 +-
 swh/storage/algos/snapshot.py                      |   2 +-
 swh/storage/backfill.py                            |   3 +-
 swh/storage/common.py                              |   9 +-
 swh/storage/in_memory.py                           |   2 +-
 swh/storage/postgresql/__init__.py                 |   0
 swh/storage/{ => postgresql}/converters.py         |  24 +-
 swh/storage/{ => postgresql}/db.py                 |   0
 swh/storage/{ => postgresql}/storage.py            |  18 +-
 swh/storage/sql/30-swh-schema.sql                  |   4 +-
 swh/storage/sql/60-swh-indexes.sql                 |  22 ++
 swh/storage/tests/algos/test_revisions_walker.py   |   2 +-
 .../tests/{test_storage.py => storage_tests.py}    |   2 +-
 swh/storage/tests/test_api_client.py               |   2 +-
 swh/storage/tests/test_in_memory.py                |   4 +-
 swh/storage/tests/test_init.py                     |   6 +-
 swh/storage/tests/test_postgresql.py               | 256 +++++++++++++++++++++
 ...converters.py => test_postgresql_converters.py} |   2 +-
 swh/storage/tests/test_revision_bw_compat.py       |   2 +-
 24 files changed, 400 insertions(+), 52 deletions(-)
 create mode 100644 sql/upgrades/160.sql
 create mode 100644 sql/upgrades/161.sql
 create mode 100644 swh/storage/postgresql/__init__.py
 rename swh/storage/{ => postgresql}/converters.py (94%)
 rename swh/storage/{ => postgresql}/db.py (100%)
 rename swh/storage/{ => postgresql}/storage.py (99%)
 rename swh/storage/tests/{test_storage.py => storage_tests.py} (99%)
 create mode 100644 swh/storage/tests/test_postgresql.py
 rename swh/storage/tests/{test_converters.py => test_postgresql_converters.py} (99%)
Changes applied before test
commit 40739076be0d54c6afe3544caf6f71324dc8b3ec
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 16:13:48 2020 +0200

    Move postgresql-related files to swh/storage/postgresql/

commit 6a53cb3ef3ad769ab015fa011e0a14702d1a14cf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 14:25:22 2020 +0200

    pg: Check revision.extra_headers is not null.
    
    Fixes a regression in 038a219f84d6b8a4f02b48f9ad3c5d823d097790,
    as it made the converter expect a list.
    
    When adding this column, we made it default to null instead of defaulting
    to an empty array, so existing records were initialized will null.
    
    This commit migrates these nulls to empty arrays, then adds a constraint
    to enforce it in the future.

commit ca3ee92da47be37c25439a49ef18c4a1d63edd58
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 14:23:54 2020 +0200

    converters: convert extra_headers to an empty list if it is None.

commit e2b14948da143925b0421415580df63e48d18a5b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 12:40:01 2020 +0200

    pg: Make date_neg_utc_offset is not null if date is not null.
    
    Fixes a regression in 038a219f84d6b8a4f02b48f9ad3c5d823d097790,
    as it made converters expect a boolean.
    
    We stopped writing this kind of nulls since we started using model
    objects for insertions, but didn't migrate existing data.
    
    This commit migrates these nulls to false, then adds a constraint
    to enforce it in the future.

commit 7dcd5708b0cffdacd0082288262ccc4d78cec2de
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 14:06:21 2020 +0200

    converters: convert neg_utc_offset to False if it is None.

commit e2f066549123f069bf0ae34d593038ffd834d9c4
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Wed Aug 19 14:54:45 2020 +0200

    backfiller: Add missing 'extra_header' field.
    
    This field wasn't backfilled; and it wasn't caught by tests, because
    revisions in swh.journal.tests.journal_data.TEST_OBJECTS were had
    an empty extra_header field.
    
    Starting with swh-journal v0.4.3, one of the revisions in TEST_OBJECTS
    has a non-empty extra_header field, so it will become a test failure
    without this commit

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/829/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/829/console

Harbormaster failed remote builds in B14682: Diff 13418!Aug 19 2020, 5:18 PM
This revision was landed with ongoing or failed builds.Aug 20 2020, 1:26 PM
Closed by commit rDSTO40739076be0d: Move postgresql-related files to swh/storage/postgresql/ (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTO40739076be0d: Move postgresql-related files to swh/storage/postgresql/.

Revision Contents
Changeset List

Diff 13418

View Options

swh/storage/__init__.py

Loading...
View Options

swh/storage/algos/diff.py

Loading...
View Options

swh/storage/algos/dir_iterators.py

Loading...
View Options

swh/storage/algos/revisions_walker.py

Loading...
View Options

swh/storage/algos/snapshot.py

Loading...
View Options

swh/storage/backfill.py

Loading...
View Options

swh/storage/common.py

Loading...
View Options

swh/storage/converters.py

Loading...
View Options

swh/storage/db.py

Loading...
View Options

swh/storage/in_memory.py

Loading...
View Options

swh/storage/postgresql/__init__.py

Loading...
View Options

swh/storage/postgresql/converters.py

Loading...
View Options

swh/storage/postgresql/db.py

Loading...
View Options

swh/storage/postgresql/storage.py

Loading...
View Options

swh/storage/storage.py

Loading...
View Options

swh/storage/tests/algos/test_revisions_walker.py

Loading...
View Options

swh/storage/tests/storage_tests.py

Loading...
View Options

swh/storage/tests/test_api_client.py

Loading...
View Options

swh/storage/tests/test_converters.py

Loading...
View Options

swh/storage/tests/test_in_memory.py

Loading...
View Options

swh/storage/tests/test_init.py

Loading...
View Options

swh/storage/tests/test_postgresql.py

Loading...
View Options

swh/storage/tests/test_postgresql_converters.py

Loading...
View Options

swh/storage/tests/test_revision_bw_compat.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
About · Developer information · Legal