Page MenuHomeSoftware Heritage
Differential D2566

Add Cassandra backend.
ClosedPublic
Actions

Authored by vlorentz on Jan 21 2020, 2:59 PM.

Details

Reviewers
douardda
ardumont
Group Reviewers
Reviewers
Commits
rDSTOeb155ad3d780: Add Cassandra backend.
Summary

Related to T2186

Diff Detail

Repository
rDSTO Storage manager
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
swh-public-ci added a comment.Jan 27 2020, 6:05 PM

Build was aborted

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/904/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/904/console

vlorentz updated this revision to Diff 9249.Jan 27 2020, 7:20 PM
vlorentz edited the summary of this revision. (Show Details)

timeout in wait_for_peer.

swh-public-ci added a comment.Jan 27 2020, 7:28 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/905/ for more details.

Harbormaster completed remote builds in B10308: Diff 9249.Jan 27 2020, 7:28 PM
vlorentz updated this revision to Diff 9251.Jan 27 2020, 7:31 PM

retrigger tests

swh-public-ci added a comment.Jan 27 2020, 7:39 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/907/ for more details.

swh-public-ci added a comment.Jan 27 2020, 7:45 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/908/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/908/console

Harbormaster failed remote builds in B10310: Diff 9251!Jan 27 2020, 7:45 PM
vlorentz updated this revision to Diff 9257.Jan 28 2020, 1:42 PM

more logs

Harbormaster failed remote builds in B10316: Diff 9257!Jan 28 2020, 1:42 PM
swh-public-ci added a comment.Jan 28 2020, 1:43 PM

Build was aborted

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/909/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/909/console

swh-public-ci added a comment.Jan 28 2020, 1:48 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/910/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/910/console

vlorentz updated this revision to Diff 9258.Jan 28 2020, 1:52 PM

fix command

swh-public-ci added a comment.Jan 28 2020, 1:53 PM

Build was aborted

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/911/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/911/console

Harbormaster failed remote builds in B10317: Diff 9258!Jan 28 2020, 1:53 PM
swh-public-ci added a comment.Jan 28 2020, 2:00 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/912/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/912/console

swh-public-ci added a comment.Jan 28 2020, 3:06 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/913/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/913/console

swh-public-ci added a comment.Jan 29 2020, 12:00 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/914/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/914/console

vlorentz updated this revision to Diff 9284.Jan 29 2020, 1:37 PM

implement endpoints missing since rebase

swh-public-ci added a comment.Jan 29 2020, 1:48 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/922/ for more details.

Harbormaster completed remote builds in B10342: Diff 9284.Jan 29 2020, 1:48 PM
swh-public-ci added a comment.Jan 29 2020, 1:49 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/923/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/923/console

vlorentz updated this revision to Diff 9287.Jan 29 2020, 1:55 PM

rebase

swh-public-ci added a comment.Jan 29 2020, 2:03 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/924/ for more details.

swh-public-ci added a comment.Jan 29 2020, 2:49 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/925/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/925/console

Harbormaster failed remote builds in B10345: Diff 9287!Jan 29 2020, 2:49 PM
vlorentz updated this revision to Diff 9291.Jan 29 2020, 3:11 PM

fix signature

swh-public-ci added a comment.Jan 29 2020, 3:18 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/928/ for more details.

swh-public-ci added a comment.Jan 29 2020, 4:02 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/929/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/929/console

Harbormaster failed remote builds in B10351: Diff 9291!Jan 29 2020, 4:02 PM
vlorentz updated this revision to Diff 9299.Jan 29 2020, 4:17 PM

Fix rebase. Depends on D2601.

vlorentz added a parent revision: D2601: Allow deprecated endpoints to be missing from a backend class..Jan 29 2020, 4:18 PM
Harbormaster failed remote builds in B10359: Diff 9299!Jan 29 2020, 4:20 PM
swh-public-ci added a comment.Jan 29 2020, 4:20 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/931/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/931/console

vlorentz updated this revision to Diff 9304.Jan 29 2020, 4:51 PM

rebase

swh-public-ci added a comment.Jan 29 2020, 5:02 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/932/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/932/console

swh-public-ci added a comment.Jan 29 2020, 5:03 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/933/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/933/console

vlorentz added a comment.EditedJan 29 2020, 5:04 PM

"Plus ça rate, plus on a de chances que ça marche !"

swh-public-ci added a comment.Jan 29 2020, 5:11 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/936/ for more details.

swh-public-ci added a comment.Jan 29 2020, 5:52 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/937/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/937/console

Harbormaster failed remote builds in B10364: Diff 9304!Jan 29 2020, 5:52 PM
swh-public-ci added a comment.Jan 29 2020, 7:29 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/938/ for more details.

Harbormaster completed remote builds in B10364: Diff 9304.Jan 29 2020, 7:29 PM
vlorentz updated this revision to Diff 9310.Jan 30 2020, 11:29 AM
  • disable debug logs
  • allocate more RAM to make tests less flaky
swh-public-ci added a comment.Jan 30 2020, 11:38 AM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/939/ for more details.

Harbormaster completed remote builds in B10370: Diff 9310.Jan 30 2020, 11:38 AM
swh-public-ci added a comment.Jan 30 2020, 12:19 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/940/ for more details.

douardda accepted this revision.Jan 31 2020, 2:09 PM
douardda added a subscriber: douardda.

Looks good to me, but it would really be nice to have a bit more documentation/explanation on how stuff work and are organized in Cassandra, be it in the code itself and as docu material in doc/

I believe the overrides of postgres specific tests is no longer needed.

Also I've a few comments.

swh/storage/cassandra/cql.py
35

What's this for?

93

No docstring at all?

124

Does it make sense not to wait a bit between read/write attempts?

This revision is now accepted and ready to land.Jan 31 2020, 2:09 PM
ardumont accepted this revision.Jan 31 2020, 2:09 PM
ardumont added a subscriber: ardumont.

Looks good.

Small remarks that can be taken care later (after the first landing ;)

I'm mostly unsure about:

  • migration. -> I guess we will have to use python script to migrate when adapting the model.
  • date precision. It seems we are losing some around the end of the diff.
swh/storage/cassandra/cql.py
112

we might be able to move the retry behavior in the retry proxy storage.
Or might be as a simple first step, decorate this method with retry behavior (storage already has the tenacity dependency)

140

statement

swh/storage/cassandra/storage.py
47

why not call it self.backend?
(we should also do the same on storage.storage)

224

<strike>one</strike> at least one

468

other endpoints no longer display the docstring, so might as well remove it as well here (and for other occurrences)
(also in master you made the necessary work for docstrings to be centralized in the storage interface ;)

vlorentz updated this revision to Diff 9341.Jan 31 2020, 2:26 PM
vlorentz marked 3 inline comments as done.

apply comments.

swh/storage/cassandra/cql.py
35

boilerplate to make python-cassandra happy. (this used to be the default config, but now it raises a warning if I don't provide it)

Basically it tells it to hit the right server directly when sending a query (TokenAwarePolicy), and if there's more than one, then pick one at random that's in the same datacenter as the client (DCAwareRoundRobinPolicy)

112

It's lower-level retries (it may be called multiple times by the same endpoint), so not exactly the same.
I'll look into tenacity.

124

it already waits for 100ms because of the timeout

swh/storage/cassandra/storage.py
47

Because we already call CassandraStorage/Storage/InMemoryStorage backends. Calling this a backend too would be confusing.

Harbormaster failed remote builds in B10402: Diff 9341!Jan 31 2020, 2:26 PM
vlorentz updated this revision to Diff 9342.Jan 31 2020, 2:28 PM

add missing comment

swh-public-ci added a comment.Jan 31 2020, 2:33 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/942/ for more details.

swh-public-ci added a comment.Jan 31 2020, 2:41 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/943/ for more details.

vlorentz updated this revision to Diff 9343.Jan 31 2020, 2:55 PM

run all tests in the samee tox environment, it wastes less time installing deps on jenkins

swh-public-ci added a comment.Jan 31 2020, 3:22 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/944/ for more details.

Harbormaster completed remote builds in B10403: Diff 9342.Jan 31 2020, 3:22 PM
swh-public-ci added a comment.Jan 31 2020, 4:04 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/945/ for more details.

Harbormaster completed remote builds in B10404: Diff 9343.Jan 31 2020, 4:04 PM
vlorentz updated this revision to Diff 9344.Jan 31 2020, 4:06 PM

rebase

vlorentz added a child revision: D2612: Add a keyspace_queue fixture to prepare keyspaces concurrently..Jan 31 2020, 4:06 PM
swh-public-ci added a comment.Jan 31 2020, 4:47 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/946/ for more details.

Harbormaster completed remote builds in B10405: Diff 9344.Jan 31 2020, 4:47 PM
Closed by commit rDSTOeb155ad3d780: Add Cassandra backend. (authored by vlorentz). · Explain WhyJan 31 2020, 5:05 PM
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDSTOeb155ad3d780: Add Cassandra backend..

Revision Contents
Changeset List

PathSize
3 lines
1 line
1 line
1 line
swh/
storage/
7 lines
cassandra/
11 lines
20 lines
72 lines
619 lines
196 lines
952 lines
tests/
260 lines
25 lines
2 lines

Diff 9346

View Options

mypy.ini

Loading...
View Options

pytest.ini

Loading...
View Options

requirements-test.txt

Loading...
View Options

requirements.txt

Loading...
View Options

swh/storage/__init__.py

Loading...
View Options

swh/storage/cassandra/__init__.py

Loading...
View Options

swh/storage/cassandra/common.py

Loading...
View Options

swh/storage/cassandra/converters.py

Loading...
View Options

swh/storage/cassandra/cql.py

Loading...
View Options

swh/storage/cassandra/schema.py

Loading...
View Options

swh/storage/cassandra/storage.py

Loading...
View Options

swh/storage/tests/test_cassandra.py

Loading...
View Options

swh/storage/tests/test_storage.py

Loading...
View Options

tox.ini

Loading...
About · Developer information · Legal