[cassandra] deploy the replaying stack
Closed, MigratedEdits Locked
Actions

Assigned To

Authored By

	vsellier
	Jul 12 2022, 12:13 PM

Description

In order to ingest data in cassandra, we need a replayer stack

The simpler is probably to deploy the storage/replayer service in kubernetes as during the test on grid5000

For the record, the configuration used for the grid5000 deployment is here:
https://forge.softwareheritage.org/source/snippets/browse/master/sysadmin/grid5000/cassandra/kubernetes/

Revisions and Commits

R260 Helm charts for swh packages
	D8499	R260:b1bf215bdd0e Make storage-replayer push their issue to sentry
	D8311	R260:3b295de813f3 Bootstrap a swh chart only with the cassandra replayer stack

Related Objects
Search...

		Status	Assigned	Task
		Migrated	gitlab-migration	T4373 [cassandra] Test the new hardware
		Migrated	gitlab-migration	T4391 [cassandra] deploy the replaying stack

Event Timeline

vsellier triaged this task as Normal priority.Jul 12 2022, 12:13 PM

vsellier created this task.

vsellier renamed this task from [cassandra] deploy the replying stack to [cassandra] deploy the replaying stack.Jul 12 2022, 3:29 PM

vsellier changed the task status from Open to Work in Progress.Aug 12 2022, 11:09 AM

vsellier mentioned this in T4373: [cassandra] Test the new hardware.

vsellier moved this task from Backlog to in-progress on the System administration board.

vsellier mentioned this in rSKCONF4434c969017a: declare the cassandra production replaying stack.Aug 19 2022, 5:38 PM

vsellier mentioned this in R260:94747dd38910: Use internal kafka addresses.Aug 19 2022, 7:32 PM

Everything is almost ready to start the ingestion tests:

The deployment of the replayers is implemented
The monitoring can also be easily deployed in a kubernetes cluster having the prometheus operator installed [1]
the grafana dashboards are also available as templates [1]

We now need to create the infrastructure that support a high number of replayers (during the test, they ran on the cassandra servers but it's not good as it can have an impact on the cassandra behavior)

[1] https://archive.softwareheritage.org/swh:1:dir:fe0c1915ac60876f06e4d76b723e04284361e7d4;origin=https://forge.softwareheritage.org/source/k8s-clusters-conf.git;visit=swh:1:snp:66a35583e901a1a5a62b4097fcd64e822316e80e;anchor=swh:1:rev:1fef5d96e81bcd55c84a759a635c6a47ec19f155;path=/production-cassandra/monitoring/

vsellier added a revision: D8311: Bootstrap a swh chart only with the cassandra replayer stack.Aug 24 2022, 6:24 PM

vsellier added a commit: R260:3b295de813f3: Bootstrap a swh chart only with the cassandra replayer stack.Aug 25 2022, 10:09 AM

vsellier mentioned this in rSPSITE81197e23e261: Remove local dns proxy of production rancher nodes.Sep 1 2022, 9:54 PM

vsellier mentioned this in rSKCONF26d8c509c2e8: Declare archive-production cluster applications.Sep 2 2022, 5:51 PM

vsellier mentioned this in rSKCONF3ebfe35fde48: Delete the production-cassandra cluster configuration.

vsellier mentioned this in rSKCONF1ad68a77cc9a: Deploy the cassandra replayers on the archive production cluster.Sep 4 2022, 7:00 PM

vsellier mentioned this in rSPRE445014116634: Declare the archive-production k8s cluster.Sep 5 2022, 3:11 PM

vsellier mentioned this in rSPRE943c46f87c61: rancher-production: Add a new node on hypervisor3 with 6 cpus.Sep 9 2022, 11:30 AM

ardumont added a revision: D8499: Make storage-replayer push their issue to sentry.Sep 19 2022, 1:47 PM

ardumont added a commit: R260:b1bf215bdd0e: Make storage-replayer push their issue to sentry.Sep 19 2022, 1:57 PM

vsellier mentioned this in R260:904535431a5a: cassandra-replay: Configure the replayer environment.Sep 19 2022, 2:46 PM