Differential D5750

cassandra: Add support for ScyllaDB
ClosedPublic
Actions

Authored by vlorentz on May 18 2021, 3:37 PM.

Details

Reviewers

douardda

Group Reviewers

Reviewers

Maniphest Tasks

T3314: Test swh.storage.cassandra with ScyllaDB

Commits

rDSTO1d880a526c48: cassandra: Add partial support for ScyllaDB

Summary

All features work but snapshot_count_branches, because ScyllaDB does not
support user-defined aggregates yet.

Migration tests hang when run after the regular tests, but I can't
figure out why. This should not be an issue for now, as we won't run
Scylla tests on the CI.

Depends on D5748.

Test Plan

It (kind of) works on my machine©

Diff Detail

Repository

rDSTO Storage manager

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 21580
Build 33528: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 33527: arc lint + arc unit

Event Timeline

vlorentz created this revision.May 18 2021, 3:37 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptMay 18 2021, 3:37 PM

Build was aborted

Patch application report for D5750 (id=20550)

Could not rebase; Attempt merge onto 53c21d4c7b...

Updating 53c21d4c..8794ec5b
Fast-forward
 swh/storage/cassandra/cql.py        | 22 ++++++++--
 swh/storage/cassandra/schema.py     | 80 +++++++++++++++++++++---------------
 swh/storage/tests/storage_tests.py  |  4 +-
 swh/storage/tests/test_cassandra.py | 82 +++++++++++++++++++++++++++----------
 4 files changed, 129 insertions(+), 59 deletions(-)

Changes applied before test

commit 8794ec5bc364db79ce9b5e43ebda6688c5d2657e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.

commit eb31e6244236fc29aeb31234af25e7b5a17ac5fd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:34:17 2021 +0200

    tests: Make test parameters order deterministic, so they don't crash pytest-xdist
    
    pytest-xdist expects the parameters to be in the same order
    in all processes.

commit 916ab07cbf18a04867857eecbad05ea1052b46b2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:33:19 2021 +0200

    test_cassandra: Improve error when the process is started but not listening

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1330/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1330/console

Harbormaster returned this revision to the author for changes because remote builds failed.May 18 2021, 4:25 PM

Harbormaster failed remote builds in B21527: Diff 20550!

Build has FAILED

Patch application report for D5750 (id=20550)

Could not rebase; Attempt merge onto 0ed4a975d1...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/cql.py        | 22 ++++++++--
 swh/storage/cassandra/schema.py     | 80 +++++++++++++++++++++---------------
 swh/storage/tests/storage_tests.py  |  4 +-
 swh/storage/tests/test_cassandra.py | 82 +++++++++++++++++++++++++++----------
 4 files changed, 129 insertions(+), 59 deletions(-)

Changes applied before test

commit 1f199582d73597fb7e3900256bf389d142e5a1d0
Merge: 0ed4a975 8794ec5b
Author: Jenkins user <jenkins@localhost>
Date:   Wed May 19 06:51:36 2021 +0000

    Merge branch 'diff-target' into HEAD

commit 8794ec5bc364db79ce9b5e43ebda6688c5d2657e
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.

commit eb31e6244236fc29aeb31234af25e7b5a17ac5fd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:34:17 2021 +0200

    tests: Make test parameters order deterministic, so they don't crash pytest-xdist
    
    pytest-xdist expects the parameters to be in the same order
    in all processes.

commit 916ab07cbf18a04867857eecbad05ea1052b46b2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:33:19 2021 +0200

    test_cassandra: Improve error when the process is started but not listening

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1331/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1331/console

fix tests

vlorentz edited the summary of this revision. (Show Details)May 19 2021, 9:26 AM

Build is green

Patch application report for D5750 (id=20558)

Could not rebase; Attempt merge onto 0ed4a975d1...

Merge made by the 'recursive' strategy.
 swh/storage/cassandra/cql.py        | 22 ++++++++--
 swh/storage/cassandra/schema.py     | 79 +++++++++++++++++++--------------
 swh/storage/tests/storage_tests.py  |  4 +-
 swh/storage/tests/test_cassandra.py | 87 +++++++++++++++++++++++++++----------
 4 files changed, 132 insertions(+), 60 deletions(-)

Changes applied before test

commit a7f4f845df3c36c60b3c4e2803bd8db064a50c4b
Merge: 0ed4a975 fe8bbea5
Author: Jenkins user <jenkins@localhost>
Date:   Wed May 19 07:31:40 2021 +0000

    Merge branch 'diff-target' into HEAD

commit fe8bbea582886b986b7a43bd1f9f5367be840b5c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add partial support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.
    
    Migration tests hang when run after the regular tests, but I can't
    figure out why. This should not be an issue for now, as we won't run
    Scylla tests on the CI.

commit eb31e6244236fc29aeb31234af25e7b5a17ac5fd
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:34:17 2021 +0200

    tests: Make test parameters order deterministic, so they don't crash pytest-xdist
    
    pytest-xdist expects the parameters to be in the same order
    in all processes.

commit 916ab07cbf18a04867857eecbad05ea1052b46b2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:33:19 2021 +0200

    test_cassandra: Improve error when the process is started but not listening

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1333/ for more details.

Harbormaster completed remote builds in B21533: Diff 20558.May 19 2021, 9:38 AM

vlorentz requested review of this revision.May 19 2021, 9:38 AM

rebase

please also add a piece of documentation somewhere on this alt. "cassandra" backend config (aka. "how to use scylladb" or something).

swh/storage/cassandra/schema.py
5	It's not clear to me: who is responsible for setting this variable? Is there a big "feature switch" somewhere else that handles this?

This revision now requires changes to proceed.May 19 2021, 10:51 AM

vlorentz added inline comments.May 19 2021, 11:10 AM

swh/storage/cassandra/schema.py
5	No, for now you need to manually change the code to use the schema.

Build is green

Patch application report for D5750 (id=20563)

Rebasing onto a92a96846b...

Current branch diff-target is up to date.

Changes applied before test

commit a9d6f7e004a0189ccbe4bd7b9df61ce31ac084c5
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add partial support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.
    
    Migration tests hang when run after the regular tests, but I can't
    figure out why. This should not be an issue for now, as we won't run
    Scylla tests on the CI.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1337/ for more details.

Harbormaster completed remote builds in B21537: Diff 20563.May 19 2021, 11:12 AM

document + add an envvar switch.

Build is green

Patch application report for D5750 (id=20596)

Rebasing onto a92a96846b...

Current branch diff-target is up to date.

Changes applied before test

commit 73588c64de7137c1a04733010e0fe438adc3f2c8
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add partial support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.
    
    Migration tests hang when run after the regular tests, but I can't
    figure out why. This should not be an issue for now, as we won't run
    Scylla tests on the CI.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1338/ for more details.

Harbormaster completed remote builds in B21561: Diff 20596.May 20 2021, 2:41 PM

consider the minor fix in the README file, otherwise ok, thanks!

README.md
222	maybe prefer 'environment variable' here (I find 'configuration variable' confusing, looks like something one has to put in the swh config file)

This revision is now accepted and ready to land.May 21 2021, 10:27 AM

vlorentz added inline comments.May 21 2021, 10:30 AM

README.md
222	yes indeed

reword

Build is green

Patch application report for D5750 (id=20614)

Rebasing onto 8e3731ace4...

First, rewinding head to replay your work on top of it...
Applying: cassandra: Add partial support for ScyllaDB

Changes applied before test

commit 041fe7b22cd115ca90544389372cc77f844889f2
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Tue May 18 15:35:41 2021 +0200

    cassandra: Add partial support for ScyllaDB
    
    All features work but snapshot_count_branches, because ScyllaDB does not
    support user-defined aggregates yet.
    
    Migration tests hang when run after the regular tests, but I can't
    figure out why. This should not be an issue for now, as we won't run
    Scylla tests on the CI.

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1340/ for more details.

Harbormaster completed remote builds in B21579: Diff 20614.May 21 2021, 11:52 AM

rebase

This revision was landed with ongoing or failed builds.May 21 2021, 12:15 PM

Closed by commit rDSTO1d880a526c48: cassandra: Add partial support for ScyllaDB (authored by vlorentz). · Explain Why

This revision was automatically updated to reflect the committed changes.

vlorentz added a commit: rDSTO1d880a526c48: cassandra: Add partial support for ScyllaDB.

Build is green

Patch application report for D5750 (id=20615)

Rebasing onto 1d880a526c...

First, rewinding head to replay your work on top of it...
Fast-forwarded diff-target to base-revision-1341-D5750.

Changes applied before test

See https://jenkins.softwareheritage.org/job/DSTO/job/tests-on-diff/1341/ for more details.

Harbormaster completed remote builds in B21580: Diff 20615.May 21 2021, 12:22 PM

vlorentz added a task: T3314: Test swh.storage.cassandra with ScyllaDB.May 26 2021, 11:24 AM

Revision Contents
Changeset List

Path

Size

Coverage (All)

Coverage (Touched)

README.md

29 lines

swh/

storage/

cassandra/

cql.py

22 lines

98%

schema.py

83 lines

81%

66%

tests/

test_cassandra.py

76 lines

90%

66%

Diff 20615

View Options

README.md

	Show First 20 Lines • Show All 115 Lines • ▼ Show 20 Lines
	a running Postgresql database.			a running Postgresql database.

	### Sample configuration			### Sample configuration

	A typical configuration `storage.yml` file is:			A typical configuration `storage.yml` file is:

	```			```
	storage:			storage:
	cls: local			cls: postgresql
	db: "dbname=softwareheritage-dev user=<user> password=<pwd>"			db: "dbname=softwareheritage-dev user=<user> password=<pwd>"
	objstorage:			objstorage:
	cls: pathslicing			cls: pathslicing
	root: /tmp/swh-storage/			root: /tmp/swh-storage/
	slicing: 0:2/2:4/4:6			slicing: 0:2/2:4/4:6
	```			```

	which means, this uses:			which means, this uses:
	▲ Show 20 Lines • Show All 49 Lines • ▼ Show 20 Lines
	configuration.			configuration.

	```			```
	storage:			storage:
	cls: remote			cls: remote
	url: http://localhost:5002/			url: http://localhost:5002/
	```			```

	You could directly define a local storage with the following snippet:			You could directly define a postgresql storage with the following snippet:

	```			```
	storage:			storage:
	cls: local			cls: postgresql
	db: service=swh-dev			db: service=swh-dev
	objstorage:			objstorage:
	cls: pathslicing			cls: pathslicing
	root: /home/storage/swh-storage/			root: /home/storage/swh-storage/
	slicing: 0:2/2:4/4:6			slicing: 0:2/2:4/4:6
	```			```

				## Cassandra

				As an alternative to PostgreSQL, swh-storage can use Cassandra as a database backend.
				It can be used like this:

				```
				storage:
				cls: cassandra
				hosts:
				- localhost
				objstorage:
				cls: pathslicing
				root: /home/storage/swh-storage/
				slicing: 0:2/2:4/4:6
				```

				The Cassandra swh-storage implementation supports both Cassandra >= 4.0-alpha2
				and ScyllaDB >= 4.4 (and possibly earlier versions, but this is untested).

				While the main code supports both transparently, running tests
				or configuring the schema requires specific code when using ScyllaDB,
				douarddaUnsubmitted Not Done Inline Actions maybe prefer 'environment variable' here (I find 'configuration variable' confusing, looks like something one has to put in the swh config file) douardda: maybe prefer 'environment variable' here (I find 'configuration variable' confusing, looks like…
				vlorentzAuthorUnsubmitted Done Inline Actions yes indeed vlorentz: yes indeed
				enabled by setting the `SWH_USE_SCYLLADB=1` environment variable.

View Options

swh/storage/cassandra/cql.py

Show First 20 Lines • Show All 974 Lines • ▼ Show 20 Lines	def origin_visit_status_get_latest(
self, origin: str, visit: int,		self, origin: str, visit: int,
) -> Optional[OriginVisitStatusRow]:		) -> Optional[OriginVisitStatusRow]:
"""Given an origin visit id, return its latest origin_visit_status		"""Given an origin visit id, return its latest origin_visit_status

"""		"""
return next(self.origin_visit_status_get(origin, visit), None)		return next(self.origin_visit_status_get(origin, visit), None)

@_prepared_select_statement(		@_prepared_select_statement(
OriginVisitStatusRow, "WHERE origin = ? AND visit = ? ORDER BY date DESC"		OriginVisitStatusRow,
		# 'visit DESC,' is optional with Cassandra 4, but ScyllaDB needs it
		"WHERE origin = ? AND visit = ? ORDER BY visit DESC, date DESC",
)		)
def origin_visit_status_get(		def origin_visit_status_get(
self, origin: str, visit: int, *, statement,		self, origin: str, visit: int, *, statement,
) -> Iterator[OriginVisitStatusRow]:		) -> Iterator[OriginVisitStatusRow]:
"""Return all origin visit statuses for a given visit		"""Return all origin visit statuses for a given visit

"""		"""
return map(		return map(
▲ Show 20 Lines • Show All 62 Lines • ▼ Show 20 Lines	) -> Iterable[RawExtrinsicMetadataRow]:
RawExtrinsicMetadataRow.from_dict,		RawExtrinsicMetadataRow.from_dict,
self._execute_with_retries(		self._execute_with_retries(
statement, [target, authority_url, after, authority_type]		statement, [target, authority_url, after, authority_type]
),		),
)		)

@_prepared_select_statement(		@_prepared_select_statement(
RawExtrinsicMetadataRow,		RawExtrinsicMetadataRow,
"WHERE target=? AND authority_type=? AND authority_url=? "		# This is equivalent to:
"AND (discovery_date, id) > (?, ?)",		# WHERE target=? AND authority_type = ? AND authority_url = ? "
		# AND (discovery_date, id) > (?, ?)"
		# but it needs to be written this way to work with ScyllaDB.
		"WHERE target=? AND (authority_type, authority_url) <= (?, ?) "
		"AND (authority_type, authority_url, discovery_date, id) > (?, ?, ?, ?)",
)		)
def raw_extrinsic_metadata_get_after_date_and_id(		def raw_extrinsic_metadata_get_after_date_and_id(
self,		self,
target: str,		target: str,
authority_type: str,		authority_type: str,
authority_url: str,		authority_url: str,
after_date: datetime.datetime,		after_date: datetime.datetime,
after_id: bytes,		after_id: bytes,
*,		*,
statement,		statement,
) -> Iterable[RawExtrinsicMetadataRow]:		) -> Iterable[RawExtrinsicMetadataRow]:
return map(		return map(
RawExtrinsicMetadataRow.from_dict,		RawExtrinsicMetadataRow.from_dict,
self._execute_with_retries(		self._execute_with_retries(
statement,		statement,
[target, authority_type, authority_url, after_date, after_id,],		[
		target,
		authority_type,
		authority_url,
		authority_type,
		authority_url,
		after_date,
		after_id,
		],
),		),
)		)

@_prepared_select_statement(		@_prepared_select_statement(
RawExtrinsicMetadataRow,		RawExtrinsicMetadataRow,
"WHERE target=? AND authority_url=? AND authority_type=?",		"WHERE target=? AND authority_url=? AND authority_type=?",
)		)
def raw_extrinsic_metadata_get(		def raw_extrinsic_metadata_get(
▲ Show 20 Lines • Show All 118 Lines • Show Last 20 Lines

View Options

swh/storage/cassandra/schema.py

# Copyright (C) 2019-2021 The Software Heritage developers		# Copyright (C) 2019-2021 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution		# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version		# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information		# See top-level LICENSE file for more information

		douarddaUnsubmitted Not Done Inline Actions It's not clear to me: who is responsible for setting this variable? Is there a big "feature switch" somewhere else that handles this? douardda: It's not clear to me: who is responsible for setting this variable? Is there a big "feature…
		vlorentzAuthorUnsubmitted Done Inline Actions No, for now you need to manually change the code to use the schema. vlorentz: No, for now you need to manually change the code to use the schema.
		import os

		_use_scylla = bool(os.environ.get("SWH_USE_SCYLLADB", ""))

		UDF_LANGUAGE = "lua" if _use_scylla else "java"

		if UDF_LANGUAGE == "java":
		# For Cassandra
CREATE_TABLES_QUERIES = [		CREATE_TABLES_QUERIES = [
"""		"""
CREATE OR REPLACE FUNCTION ascii_bins_count_sfunc (		CREATE OR REPLACE FUNCTION ascii_bins_count_sfunc (
state tuple<int, map<ascii, int>>, -- (nb_none, map<target_type, nb>)		state tuple<int, map<ascii, int>>, -- (nb_none, map<target_type, nb>)
bin_name ascii		bin_name ascii
)		)
CALLED ON NULL INPUT		CALLED ON NULL INPUT
RETURNS tuple<int, map<ascii, int>>		RETURNS tuple<int, map<ascii, int>>
LANGUAGE java AS		LANGUAGE java AS
$$		$$
if (bin_name == null) {		if (bin_name == null) {
state.setInt(0, state.getInt(0) + 1);		state.setInt(0, state.getInt(0) + 1);
}		}
else {		else {
Map<String, Integer> counters = state.getMap(		Map<String, Integer> counters = state.getMap(
1, String.class, Integer.class);		1, String.class, Integer.class);
Integer nb = counters.get(bin_name);		Integer nb = counters.get(bin_name);
if (nb == null) {		if (nb == null) {
nb = 0;		nb = 0;
}		}
counters.put(bin_name, nb + 1);		counters.put(bin_name, nb + 1);
state.setMap(1, counters, String.class, Integer.class);		state.setMap(1, counters, String.class, Integer.class);
}		}
return state;		return state;
$$		$$;""",
;""",
"""		"""
CREATE OR REPLACE AGGREGATE ascii_bins_count ( ascii )		CREATE OR REPLACE AGGREGATE ascii_bins_count ( ascii )
SFUNC ascii_bins_count_sfunc		SFUNC ascii_bins_count_sfunc
STYPE tuple<int, map<ascii, int>>		STYPE tuple<int, map<ascii, int>>
INITCOND (0, {})		INITCOND (0, {})
;""",		;""",
		]
		elif UDF_LANGUAGE == "lua":
		# For ScyllaDB
		# TODO: this is not implementable yet, because ScyllaDB does not support
		# user-defined aggregates. https://github.com/scylladb/scylla/issues/7201
		CREATE_TABLES_QUERIES = []
		else:
		assert False, f"{UDF_LANGUAGE} must be 'lua' or 'java'"

		CREATE_TABLES_QUERIES = [
		*CREATE_TABLES_QUERIES,
"""		"""
CREATE TYPE IF NOT EXISTS microtimestamp (		CREATE TYPE IF NOT EXISTS microtimestamp (
seconds bigint,		seconds bigint,
microseconds int		microseconds int
);""",		);""",
"""		"""
CREATE TYPE IF NOT EXISTS microtimestamp_with_timezone (		CREATE TYPE IF NOT EXISTS microtimestamp_with_timezone (
timestamp frozen<microtimestamp>,		timestamp frozen<microtimestamp>,
▲ Show 20 Lines • Show All 110 Lines • ▼ Show 20 Lines	CREATE TABLE IF NOT EXISTS origin_visit_status (
origin text,		origin text,
visit bigint,		visit bigint,
date timestamp,		date timestamp,
type text,		type text,
status ascii,		status ascii,
metadata text,		metadata text,
snapshot blob,		snapshot blob,
PRIMARY KEY ((origin), visit, date)		PRIMARY KEY ((origin), visit, date)
);""",		)
		WITH CLUSTERING ORDER BY (visit DESC, date DESC)
		;""", # 'WITH CLUSTERING ORDER BY' is optional with Cassandra 4, but ScyllaDB needs it
"""		"""
CREATE TABLE IF NOT EXISTS origin (		CREATE TABLE IF NOT EXISTS origin (
sha1 blob PRIMARY KEY,		sha1 blob PRIMARY KEY,
url text,		url text,
next_visit_id int,		next_visit_id int,
-- We need integer visit ids for compatibility with the pgsql		-- We need integer visit ids for compatibility with the pgsql
-- storage, so we're using lightweight transactions with this trick:		-- storage, so we're using lightweight transactions with this trick:
-- https://stackoverflow.com/a/29391877/539465		-- https://stackoverflow.com/a/29391877/539465
▲ Show 20 Lines • Show All 142 Lines • Show Last 20 Lines

View Options

swh/storage/tests/test_cassandra.py

	# Copyright (C) 2018-2021 The Software Heritage developers			# Copyright (C) 2018-2021 The Software Heritage developers
	# See the AUTHORS file at the top-level directory of this distribution			# See the AUTHORS file at the top-level directory of this distribution
	# License: GNU General Public License version 3, or any later version			# License: GNU General Public License version 3, or any later version
	# See top-level LICENSE file for more information			# See top-level LICENSE file for more information

	import datetime			import datetime
	import itertools			import itertools
	import os			import os
				import resource
	import signal			import signal
	import socket			import socket
	import subprocess			import subprocess
	import time			import time
	from typing import Any, Dict			from typing import Any, Dict

	import attr			import attr
	import pytest			import pytest
	Show All 36 Lines

	# speed-up by disabling period saving to disk			# speed-up by disabling period saving to disk
	key_cache_save_period: 0			key_cache_save_period: 0
	row_cache_save_period: 0			row_cache_save_period: 0
	trickle_fsync: false			trickle_fsync: false
	commitlog_sync_period_in_ms: 100000			commitlog_sync_period_in_ms: 100000
	"""			"""

				SCYLLA_EXTRA_CONFIG_TEMPLATE = """
				experimental_features:
				- udf
				view_hints_directory: {data_dir}/view_hints
				prometheus_port: 0 # disable prometheus server
				start_rpc: false # disable thrift server
				api_port: {api_port}
				"""


	def free_port():			def free_port():
	sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)			sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
	sock.bind(("127.0.0.1", 0))			sock.bind(("127.0.0.1", 0))
	port = sock.getsockname()[1]			port = sock.getsockname()[1]
	sock.close()			sock.close()
	return port			return port


	def wait_for_peer(addr, port):			def wait_for_peer(addr, port):
	wait_until = time.time() + 20			wait_until = time.time() + 60
	while time.time() < wait_until:			while time.time() < wait_until:
	try:			try:
	sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)			sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
	sock.connect((addr, port))			sock.connect((addr, port))
	except ConnectionRefusedError:			except ConnectionRefusedError:
	time.sleep(0.1)			time.sleep(0.1)
	else:			else:
	sock.close()			sock.close()
	return True			return True
	return False			return False


	@pytest.fixture(scope="session")			@pytest.fixture(scope="session")
	def cassandra_cluster(tmpdir_factory):			def cassandra_cluster(tmpdir_factory):
	cassandra_conf = tmpdir_factory.mktemp("cassandra_conf")			cassandra_conf = tmpdir_factory.mktemp("cassandra_conf")
	cassandra_data = tmpdir_factory.mktemp("cassandra_data")			cassandra_data = tmpdir_factory.mktemp("cassandra_data")
	cassandra_log = tmpdir_factory.mktemp("cassandra_log")			cassandra_log = tmpdir_factory.mktemp("cassandra_log")
	native_transport_port = free_port()			native_transport_port = free_port()
	storage_port = free_port()			storage_port = free_port()
	jmx_port = free_port()			jmx_port = free_port()
				api_port = free_port()

				use_scylla = bool(os.environ.get("SWH_USE_SCYLLADB", ""))

				cassandra_bin = os.environ.get(
				"SWH_CASSANDRA_BIN", "/usr/bin/scylla" if use_scylla else "/usr/sbin/cassandra"
				)

	with open(str(cassandra_conf.join("cassandra.yaml")), "w") as fd:			if use_scylla:
				os.makedirs(cassandra_conf.join("conf"))
				config_path = cassandra_conf.join("conf/scylla.yaml")
				config_template = CONFIG_TEMPLATE + SCYLLA_EXTRA_CONFIG_TEMPLATE
				else:
				config_path = cassandra_conf.join("cassandra.yaml")
				config_template = CONFIG_TEMPLATE

				with open(str(config_path), "w") as fd:
	fd.write(			fd.write(
	CONFIG_TEMPLATE.format(			config_template.format(
	data_dir=str(cassandra_data),			data_dir=str(cassandra_data),
	storage_port=storage_port,			storage_port=storage_port,
	native_transport_port=native_transport_port,			native_transport_port=native_transport_port,
				api_port=api_port,
	)			)
	)			)

	if os.environ.get("SWH_CASSANDRA_LOG"):			if os.environ.get("SWH_CASSANDRA_LOG"):
	stdout = stderr = None			stdout = stderr = None
	else:			else:
	stdout = stderr = subprocess.DEVNULL			stdout = stderr = subprocess.DEVNULL

	cassandra_bin = os.environ.get("SWH_CASSANDRA_BIN", "/usr/sbin/cassandra")
	env = {			env = {
	"MAX_HEAP_SIZE": "300M",			"MAX_HEAP_SIZE": "300M",
	"HEAP_NEWSIZE": "50M",			"HEAP_NEWSIZE": "50M",
	"JVM_OPTS": "-Xlog:gc=error:file=%s/gc.log" % cassandra_log,			"JVM_OPTS": "-Xlog:gc=error:file=%s/gc.log" % cassandra_log,
	}			}
	if "JAVA_HOME" in os.environ:			if "JAVA_HOME" in os.environ:
	env["JAVA_HOME"] = os.environ["JAVA_HOME"]			env["JAVA_HOME"] = os.environ["JAVA_HOME"]

				if use_scylla:
				env = {
				**env,
				"SCYLLA_HOME": cassandra_conf,
				}
				# prevent "NOFILE rlimit too low (recommended setting 200000,
				# minimum setting 10000; refusing to start."
				resource.setrlimit(resource.RLIMIT_NOFILE, (200000, 200000))

				proc = subprocess.Popen(
				[cassandra_bin, "--developer-mode=1",],
				start_new_session=True,
				env=env,
				stdout=stdout,
				stderr=stderr,
				)
				else:
	proc = subprocess.Popen(			proc = subprocess.Popen(
	[			[
	cassandra_bin,			cassandra_bin,
	"-Dcassandra.config=file://%s/cassandra.yaml" % cassandra_conf,			"-Dcassandra.config=file://%s/cassandra.yaml" % cassandra_conf,
	"-Dcassandra.logdir=%s" % cassandra_log,			"-Dcassandra.logdir=%s" % cassandra_log,
	"-Dcassandra.jmx.local.port=%d" % jmx_port,			"-Dcassandra.jmx.local.port=%d" % jmx_port,
	"-Dcassandra-foreground=yes",			"-Dcassandra-foreground=yes",
	],			],
	start_new_session=True,			start_new_session=True,
	env=env,			env=env,
	stdout=stdout,			stdout=stdout,
	stderr=stderr,			stderr=stderr,
	)			)

	listening = wait_for_peer("127.0.0.1", native_transport_port)			listening = wait_for_peer("127.0.0.1", native_transport_port)

	if listening:			if listening:
	yield (["127.0.0.1"], native_transport_port)			yield (["127.0.0.1"], native_transport_port)

	if not listening or os.environ.get("SWH_CASSANDRA_LOG"):			if not listening or os.environ.get("SWH_CASSANDRA_LOG"):
	debug_log_path = str(cassandra_log.join("debug.log"))			debug_log_path = str(cassandra_log.join("debug.log"))
	▲ Show 20 Lines • Show All 500 Lines • Show Last 20 Lines

cassandra: Add support for ScyllaDBClosedPublicActions

Details

Diff Detail

Event Timeline

Patch application report for D5750 (id=20550)

Changes applied before test

Patch application report for D5750 (id=20550)

Changes applied before test

Patch application report for D5750 (id=20558)

Changes applied before test

Patch application report for D5750 (id=20563)

Changes applied before test

Patch application report for D5750 (id=20596)

Changes applied before test

Patch application report for D5750 (id=20614)

Changes applied before test

Patch application report for D5750 (id=20615)

Changes applied before test

Revision ContentsChangeset List

Diff 20615

README.md

swh/storage/cassandra/cql.py

swh/storage/cassandra/schema.py

swh/storage/tests/test_cassandra.py

cassandra: Add support for ScyllaDB
ClosedPublic
Actions

Revision Contents
Changeset List