Page MenuHomeSoftware Heritage
Differential D5750

cassandra: Add support for ScyllaDB
ClosedPublic
Actions

Authored by vlorentz on May 18 2021, 3:37 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Maniphest Tasks
T3314: Test swh.storage.cassandra with ScyllaDB
Commits
rDSTO1d880a526c48: cassandra: Add partial support for ScyllaDB
Summary

All features work but snapshot_count_branches, because ScyllaDB does not
support user-defined aggregates yet.

Migration tests hang when run after the regular tests, but I can't
figure out why. This should not be an issue for now, as we won't run
Scylla tests on the CI.

Depends on D5748.

Test Plan

It (kind of) works on my machine©

Diff Detail

Event Timeline

vlorentz created this revision.May 18 2021, 3:37 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMay 18 2021, 3:37 PM
swh-public-ci added a comment.May 18 2021, 4:25 PM

Build was aborted

Patch application report for D5750 (id=20550)

Could not rebase; Attempt merge onto 53c21d4c7b...

Updating 53c21d4c..8794ec5b
Fast-forward
 swh/storage/cassandra/cql.py        | 22 ++++++++--
 swh/storage/cassandra/schema.py     | 80 +++++++++++++++++++++---------------
 swh/storage/tests/storage_tests.py  |  4 +-
 swh/storage/tests/test_cassandra.py | 82 +++++++++++++++++++++++++++----------
 4 files changed, 129 insertions(+), 59 deletions(-)
Changes applied before test
commit 8794ec5bc364db79ce9b5e43ebda6688c5d2657e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.

commit eb31e6244236fc29aeb31234af25e7b5a17ac5fd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:34:17 2021 +0200

    tests: Make test parameters order deterministic, so they don't crash pytest-xdist
    
    pytest-xdist expects the parameters to be in the same order
    in all processes.

commit 916ab07cbf18a04867857eecbad05ea1052b46b2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:33:19 2021 +0200

    test_cassandra: Improve error when the process is started but not listening

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1330/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1330/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 18 2021, 4:25 PM
Harbormaster failed remote builds in B21527: Diff 20550!
swh-public-ci added a comment.May 19 2021, 8:58 AM

Build has FAILED

Patch application report for D5750 (id=20550)

Could not rebase; Attempt merge onto 0ed4a975d1...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/cql.py        | 22 ++++++++--
 swh/storage/cassandra/schema.py     | 80 +++++++++++++++++++++---------------
 swh/storage/tests/storage_tests.py  |  4 +-
 swh/storage/tests/test_cassandra.py | 82 +++++++++++++++++++++++++++----------
 4 files changed, 129 insertions(+), 59 deletions(-)
Changes applied before test
commit 1f199582d73597fb7e3900256bf389d142e5a1d0
Merge: 0ed4a975 8794ec5b
Author: Jenkins user <jenkins@localhost>
Date:   Wed May 19 06:51:36 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 8794ec5bc364db79ce9b5e43ebda6688c5d2657e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.

commit eb31e6244236fc29aeb31234af25e7b5a17ac5fd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:34:17 2021 +0200

    tests: Make test parameters order deterministic, so they don't crash pytest-xdist
    
    pytest-xdist expects the parameters to be in the same order
    in all processes.

commit 916ab07cbf18a04867857eecbad05ea1052b46b2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:33:19 2021 +0200

    test_cassandra: Improve error when the process is started but not listening

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1331/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1331/console

vlorentz updated this revision to Diff 20558.May 19 2021, 9:26 AM

fix tests

vlorentz edited the summary of this revision. (Show Details)May 19 2021, 9:26 AM
swh-public-ci added a comment.May 19 2021, 9:38 AM

Build is green

Patch application report for D5750 (id=20558)

Could not rebase; Attempt merge onto 0ed4a975d1...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/cql.py        | 22 ++++++++--
 swh/storage/cassandra/schema.py     | 79 +++++++++++++++++++--------------
 swh/storage/tests/storage_tests.py  |  4 +-
 swh/storage/tests/test_cassandra.py | 87 +++++++++++++++++++++++++++----------
 4 files changed, 132 insertions(+), 60 deletions(-)
Changes applied before test
commit a7f4f845df3c36c60b3c4e2803bd8db064a50c4b
Merge: 0ed4a975 fe8bbea5
Author: Jenkins user <jenkins@localhost>
Date:   Wed May 19 07:31:40 2021 +0000

    Merge branch 'diff-target' into HEAD

commit fe8bbea582886b986b7a43bd1f9f5367be840b5c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add partial support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.
    
    Migration tests hang when run after the regular tests, but I can't
    figure out why. This should not be an issue for now, as we won't run
    Scylla tests on the CI.

commit eb31e6244236fc29aeb31234af25e7b5a17ac5fd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:34:17 2021 +0200

    tests: Make test parameters order deterministic, so they don't crash pytest-xdist
    
    pytest-xdist expects the parameters to be in the same order
    in all processes.

commit 916ab07cbf18a04867857eecbad05ea1052b46b2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:33:19 2021 +0200

    test_cassandra: Improve error when the process is started but not listening

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1333/ for more details.

Harbormaster completed remote builds in B21533: Diff 20558.May 19 2021, 9:38 AM
vlorentz requested review of this revision.May 19 2021, 9:38 AM
vlorentz updated this revision to Diff 20563.May 19 2021, 10:49 AM

rebase

douardda requested changes to this revision.May 19 2021, 10:51 AM
douardda added a subscriber: douardda.

please also add a piece of documentation somewhere on this alt. "cassandra" backend config (aka. "how to use scylladb" or something).

swh/storage/cassandra/schema.py
5

It's not clear to me: who is responsible for setting this variable? Is there a big "feature switch" somewhere else that handles this?

This revision now requires changes to proceed.May 19 2021, 10:51 AM
vlorentz added inline comments.May 19 2021, 11:10 AM
swh/storage/cassandra/schema.py
5

No, for now you need to manually change the code to use the schema.

swh-public-ci added a comment.May 19 2021, 11:12 AM

Build is green

Patch application report for D5750 (id=20563)

Rebasing onto a92a96846b...

Current branch diff-target is up to date.
Changes applied before test
commit a9d6f7e004a0189ccbe4bd7b9df61ce31ac084c5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add partial support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.
    
    Migration tests hang when run after the regular tests, but I can't
    figure out why. This should not be an issue for now, as we won't run
    Scylla tests on the CI.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1337/ for more details.

Harbormaster completed remote builds in B21537: Diff 20563.May 19 2021, 11:12 AM
vlorentz updated this revision to Diff 20596.May 20 2021, 2:33 PM

document + add an envvar switch.

swh-public-ci added a comment.May 20 2021, 2:41 PM

Build is green

Patch application report for D5750 (id=20596)

Rebasing onto a92a96846b...

Current branch diff-target is up to date.
Changes applied before test
commit 73588c64de7137c1a04733010e0fe438adc3f2c8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add partial support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.
    
    Migration tests hang when run after the regular tests, but I can't
    figure out why. This should not be an issue for now, as we won't run
    Scylla tests on the CI.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1338/ for more details.

Harbormaster completed remote builds in B21561: Diff 20596.May 20 2021, 2:41 PM
douardda accepted this revision.May 21 2021, 10:27 AM

consider the minor fix in the README file, otherwise ok, thanks!

README.md
223 ↗(On Diff #20596)

maybe prefer 'environment variable' here (I find 'configuration variable' confusing, looks like something one has to put in the swh config file)

This revision is now accepted and ready to land.May 21 2021, 10:27 AM
vlorentz added inline comments.May 21 2021, 10:30 AM
README.md
223 ↗(On Diff #20596)

yes indeed

vlorentz updated this revision to Diff 20614.May 21 2021, 11:44 AM

reword

swh-public-ci added a comment.May 21 2021, 11:52 AM

Build is green

Patch application report for D5750 (id=20614)

Rebasing onto 8e3731ace4...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Add partial support for ScyllaDB
Changes applied before test
commit 041fe7b22cd115ca90544389372cc77f844889f2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add partial support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.
    
    Migration tests hang when run after the regular tests, but I can't
    figure out why. This should not be an issue for now, as we won't run
    Scylla tests on the CI.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1340/ for more details.

Harbormaster completed remote builds in B21579: Diff 20614.May 21 2021, 11:52 AM
vlorentz updated this revision to Diff 20615.May 21 2021, 12:14 PM

rebase

This revision was landed with ongoing or failed builds.May 21 2021, 12:15 PM
Closed by commit rDSTO1d880a526c48: cassandra: Add partial support for ScyllaDB (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTO1d880a526c48: cassandra: Add partial support for ScyllaDB.
swh-public-ci added a comment.May 21 2021, 12:22 PM

Build is green

Patch application report for D5750 (id=20615)

Rebasing onto 1d880a526c...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-1341-D5750.
Changes applied before test



See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1341/ for more details.
Harbormaster completed remote builds in B21580: Diff 20615.May 21 2021, 12:22 PM
vlorentz added a task: T3314: Test swh.storage.cassandra with ScyllaDB.May 26 2021, 11:24 AM
Revision Contents
Changeset List
PathSizeCoverage (All)Coverage (Touched)
swh/
storage/
cassandra/
22 lines98%
Loading...
79 lines78%
Loading...
tests/
74 lines90%
Loading...
Diff 20563
View Options
swh/storage/cassandra/cql.py
Loading...
View Options
swh/storage/cassandra/schema.py
Loading...
View Options
swh/storage/tests/test_cassandra.py
Loading...
About · Developer information · Legal