Page MenuHomeSoftware Heritage
Differential D5584

cassandra: Add a test of a 'complex' migration, with a PK update
ClosedPublic
Actions

Authored by vlorentz on Apr 23 2021, 10:43 AM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T2602: Investigate how to upgrade the schema of the Cassandra storage
Commits
rDSTO051b77152356: cassandra: Add a test of a 'complex' migration, with a PK update
Summary

Depends on D5582

Diff Detail

Unit TestsFailed
View All

TimeTest
30 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_content_add_collision
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f62477c6b00> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f6247ba4f98>
28 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_content_add_metadata_collision
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f62484c4a58> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f624b85b3c8>
29 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_content_add_twice
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f62477835f8> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f62477c6470>
30 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_directory_add_twice
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f62483fd550> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f62479c27b8>
28 msJenkins > .tox.py3.lib.python3.7.site-packages.swh.storage.tests.test_api_client.TestStorageApi::test_extid_add_extid_multicity
self = <swh.storage.tests.test_api_client.TestStorageApi object at 0x7f624798a208> swh_storage = <RemoteStorage url=mock://example.com/> sample_data = <swh.storage.tests.storage_data.StorageData object at 0x7f6247299da0>
View Full Test Results (43 Failed · 771 Passed · 23 Skipped)

Event Timeline

vlorentz created this revision.Apr 23 2021, 10:43 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 23 2021, 10:43 AM
swh-public-ci added a comment.Apr 23 2021, 10:49 AM

Build is green

Patch application report for D5584 (id=19953)

Could not rebase; Attempt merge onto eb8c147a54...

Updating eb8c147a..91112860
Fast-forward
 swh/storage/cassandra/model.py                |   7 +-
 swh/storage/cassandra/storage.py              | 145 +++++++----
 swh/storage/in_memory.py                      |   1 +
 swh/storage/tests/test_cassandra_migration.py | 341 ++++++++++++++++++++++++++
 4 files changed, 441 insertions(+), 53 deletions(-)
 create mode 100644 swh/storage/tests/test_cassandra_migration.py
Changes applied before test
commit 91112860b9fd144a66a25960387d18394cda162b
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 23 10:43:27 2021 +0200

    cassandra: Add a test of a 'complex' migration, with a PK update

commit 879b240c123dc47afffc6bcb3840be32d6bf1eab
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1268/ for more details.

Harbormaster completed remote builds in B20955: Diff 19953.Apr 23 2021, 10:49 AM
vlorentz requested review of this revision.Apr 23 2021, 10:49 AM
vlorentz added a task: T2602: Investigate how to upgrade the schema of the Cassandra storage.Apr 26 2021, 10:25 AM
douardda accepted this revision.Apr 27 2021, 10:34 AM
douardda added a subscriber: douardda.

I would not say all this is crystal clear to me, but overall looks fine.

This revision is now accepted and ready to land.Apr 27 2021, 10:34 AM
vlorentz updated this revision to Diff 20127.Apr 29 2021, 10:18 AM

rebase + rename 'check_missing' to 'not allow_overwrite'

swh-public-ci added a comment.Apr 29 2021, 10:28 AM

Build has FAILED

Patch application report for D5584 (id=20127)

Could not rebase; Attempt merge onto 98804f9e12...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/model.py                |   7 +-
 swh/storage/cassandra/storage.py              | 148 +++++++----
 swh/storage/in_memory.py                      |   1 +
 swh/storage/tests/test_cassandra_migration.py | 341 ++++++++++++++++++++++++++
 4 files changed, 444 insertions(+), 53 deletions(-)
 create mode 100644 swh/storage/tests/test_cassandra_migration.py
Changes applied before test
commit 7a608c3f3cf7a215d97aae04ddbe5a5d69921d98
Merge: 98804f9e fe64cab5
Author: Jenkins user <jenkins@localhost>
Date:   Thu Apr 29 08:23:26 2021 +0000

    Merge branch 'diff-target' into HEAD

commit fe64cab5d93a861290db8e993ede93c34fb41360
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 23 10:43:27 2021 +0200

    cassandra: Add a test of a 'complex' migration, with a PK update

commit 25bc523062eba8b5d75f9dc96746cd339e29ce6e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1280/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1280/console

Harbormaster failed remote builds in B21118: Diff 20127!Apr 29 2021, 10:28 AM
vlorentz updated this revision to Diff 20130.Apr 29 2021, 10:57 AM

rebase

swh-public-ci added a comment.Apr 29 2021, 11:06 AM

Build has FAILED

Patch application report for D5584 (id=20130)

Could not rebase; Attempt merge onto 98804f9e12...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/model.py                |   7 +-
 swh/storage/cassandra/storage.py              | 148 +++++++----
 swh/storage/in_memory.py                      |   1 +
 swh/storage/tests/test_cassandra_migration.py | 341 ++++++++++++++++++++++++++
 4 files changed, 444 insertions(+), 53 deletions(-)
 create mode 100644 swh/storage/tests/test_cassandra_migration.py
Changes applied before test
commit c644e62433eae3cdfc8e395b95e9b2d352f8f5c4
Merge: 98804f9e ef1a018a
Author: Jenkins user <jenkins@localhost>
Date:   Thu Apr 29 09:02:04 2021 +0000

    Merge branch 'diff-target' into HEAD

commit ef1a018adce2e3e9ea72109f7bcc41b4a38fb2bf
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 23 10:43:27 2021 +0200

    cassandra: Add a test of a 'complex' migration, with a PK update

commit 6edad20297793e2fed9fbd3f8042f4a7741d90ba
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1282/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1282/console

Harbormaster failed remote builds in B21121: Diff 20130!Apr 29 2021, 11:06 AM
vlorentz updated this revision to Diff 20139.Apr 29 2021, 12:24 PM

rebase

swh-public-ci added a comment.Apr 29 2021, 12:38 PM

Build has FAILED

Patch application report for D5584 (id=20139)

Could not rebase; Attempt merge onto 98804f9e12...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/model.py                |   7 +-
 swh/storage/cassandra/storage.py              | 150 +++++++----
 swh/storage/in_memory.py                      |   1 +
 swh/storage/tests/test_cassandra.py           | 109 +++++++-
 swh/storage/tests/test_cassandra_migration.py | 341 ++++++++++++++++++++++++++
 5 files changed, 553 insertions(+), 55 deletions(-)
 create mode 100644 swh/storage/tests/test_cassandra_migration.py
Changes applied before test
commit fc1f64022568e41fe8cc853c8f6ddce562b30f5f
Merge: 98804f9e f189ac34
Author: Jenkins user <jenkins@localhost>
Date:   Thu Apr 29 10:31:41 2021 +0000

    Merge branch 'diff-target' into HEAD

commit f189ac34b0367f256eec323bb52166660f390a4a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 23 10:43:27 2021 +0200

    cassandra: Add a test of a 'complex' migration, with a PK update

commit 7567146f7b712b2c1b27f61682e391f120f276d0
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1284/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1284/console

Harbormaster failed remote builds in B21129: Diff 20139!Apr 29 2021, 12:38 PM
vlorentz updated this revision to Diff 20145.Apr 29 2021, 2:15 PM

fix doc

swh-public-ci added a comment.Apr 29 2021, 2:30 PM

Build is green

Patch application report for D5584 (id=20145)

Could not rebase; Attempt merge onto 98804f9e12...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/model.py                |   7 +-
 swh/storage/cassandra/storage.py              | 152 ++++++++----
 swh/storage/in_memory.py                      |   1 +
 swh/storage/tests/test_cassandra.py           | 109 +++++++-
 swh/storage/tests/test_cassandra_migration.py | 341 ++++++++++++++++++++++++++
 5 files changed, 555 insertions(+), 55 deletions(-)
 create mode 100644 swh/storage/tests/test_cassandra_migration.py
Changes applied before test
commit 84bf14b27c8118ac2da240a2a03ea3d35e5d8d9e
Merge: 98804f9e df4bff6f
Author: Jenkins user <jenkins@localhost>
Date:   Thu Apr 29 12:23:00 2021 +0000

    Merge branch 'diff-target' into HEAD

commit df4bff6f5614a34a4936326a0c0ca5b93383a55f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 23 10:43:27 2021 +0200

    cassandra: Add a test of a 'complex' migration, with a PK update

commit 5a0915c569fd9db649eaaea14e02a32c8aec1909
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1288/ for more details.

Harbormaster completed remote builds in B21134: Diff 20145.Apr 29 2021, 2:30 PM
vlorentz updated this revision to Diff 20231.May 3 2021, 5:41 PM

rebase

swh-public-ci added a comment.May 3 2021, 5:55 PM

Build is green

Patch application report for D5584 (id=20231)

Could not rebase; Attempt merge onto 92d551a4a5...

Updating 92d551a4..051b7715
Fast-forward
 swh/storage/cassandra/model.py                |   7 +-
 swh/storage/cassandra/storage.py              | 152 ++++++++----
 swh/storage/in_memory.py                      |   1 +
 swh/storage/tests/test_cassandra.py           | 109 +++++++-
 swh/storage/tests/test_cassandra_migration.py | 341 ++++++++++++++++++++++++++
 5 files changed, 555 insertions(+), 55 deletions(-)
 create mode 100644 swh/storage/tests/test_cassandra_migration.py
Changes applied before test
commit 051b771523567da7fc70b7937eb3d1a6ec27a85d
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Fri Apr 23 10:43:27 2021 +0200

    cassandra: Add a test of a 'complex' migration, with a PK update

commit f23346144dd3032df0664b3a49b8fe254aed217f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 22 20:27:33 2021 +0200

    cassandra: Add 'check_missing' option, to allow updating objects
    
    as part of a migration.
    
    Also write a first test that simulates how a simple migration would go.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1298/ for more details.

Harbormaster completed remote builds in B21219: Diff 20231.May 3 2021, 5:56 PM
Closed by commit rDSTO051b77152356: cassandra: Add a test of a 'complex' migration, with a PK update (authored by vlorentz). · Explain WhyMay 4 2021, 10:59 AM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTO051b77152356: cassandra: Add a test of a 'complex' migration, with a PK update.

Revision Contents
Changeset List

PathSize
swh/
storage/
cassandra/
7 lines
tests/
184 lines

Diff 20130

View Options

swh/storage/cassandra/model.py

Loading...
View Options

swh/storage/tests/test_cassandra_migration.py

Loading...
About · Developer information · Legal