No OneTemporary
Actions

Size

25 KB

Subscribers

None

View Options

	diff --git a/sysadmin/grid5000/cassandra/.gitignore b/sysadmin/grid5000/cassandra/.gitignore
	index 9de872b..970fc72 100644
	--- a/sysadmin/grid5000/cassandra/.gitignore
	+++ b/sysadmin/grid5000/cassandra/.gitignore
	@@ -1,4 +1,6 @@
	.terraform
	terraform.tfstate*
	.vagrant
	playbook.retry
	+.credentials
	+.vault_password
	diff --git a/sysadmin/grid5000/cassandra/01-run.sh b/sysadmin/grid5000/cassandra/01-run.sh
	index 5fdf4a3..8667da6 100755
	--- a/sysadmin/grid5000/cassandra/01-run.sh
	+++ b/sysadmin/grid5000/cassandra/01-run.sh
	@@ -1,26 +1,28 @@
	#!/usr/bin/env bash

	set -eu

	SCRIPT_DIR="$(pwd $(dirname @0))"

	source "${SCRIPT_DIR}/environment.cfg"

	+rm -f nodes.installed
	+
	${SCRIPT_DIR}/02-reserve_nodes.sh

	echo "########### Waiting for node installations"
	while [ ! -e ${SCRIPT_DIR}/nodes.installed ]; do
	sleep 2
	done
	echo "########### Node installations done"

	echo "########### Initialize cassandra"
	FIRST_STORAGE_HOST="$(echo ${STORAGE_HOSTS} \| cut -f1 -d' ')"
	STORAGE_NODE="${FIRST_STORAGE_HOST}.${G5K_SITE}"

	ssh "${SSH_USER}@${STORAGE_NODE}" /usr/local/bin/swh-storage-init-cassandra.sh

	echo "####### FINISHED"

	echo "####### Sleeping"
	sleep infinity
	diff --git a/sysadmin/grid5000/cassandra/03-deploy_nodes.sh b/sysadmin/grid5000/cassandra/03-deploy_nodes.sh
	index 66dabdb..a947f9f 100755
	--- a/sysadmin/grid5000/cassandra/03-deploy_nodes.sh
	+++ b/sysadmin/grid5000/cassandra/03-deploy_nodes.sh
	@@ -1,33 +1,33 @@
	#!/usr/bin/env bash

	set -eu

	SCRIPT_DIR="$(pwd $(dirname @0))"

	source "${SCRIPT_DIR}/environment.cfg"

	echo "########### Nodes:"
	uniq "${OAR_FILE_NODES}"
	echo "########### Installing os on nodes"

	INSTALLED_OS_STAMP="${OAR_JOB_ID}.os.stamp"

	if [ ! -e "${SCRIPT_DIR}/${INSTALLED_OS_STAMP}" ]; then
	${SCRIPT_DIR}/_install_os.sh
	touch "${SCRIPT_DIR}/${INSTALLED_OS_STAMP}"
	fi

	uniq "${OAR_NODE_FILE}" > ${SCRIPT_DIR}/nodes.lst

	echo "${CASSANDRA_HOSTS}" \| sed 's/ /,/' > ${SCRIPT_DIR}/cassandra_seeds.lst

	-parallel --halt now,fail=1 rsync -avP . "${SSH_USER}"@{}:install < ${SCRIPT_DIR}/nodes.lst
	+time parallel --halt now,fail=1 rsync -avP . "${SSH_USER}"@{}:install < ${SCRIPT_DIR}/nodes.lst

	time parallel --halt now,fail=1 -u ssh ${SSH_OPTIONS} "${SSH_USER}"@{} install/_provision_node.sh < ${SCRIPT_DIR}/nodes.lst

	echo "########### Cassandra installation done"
	touch ${SCRIPT_DIR}/nodes.installed

	# The script must not exit to avoid the oar job to be killed
	echo "########### Sleeping"
	sleep infinity
	diff --git a/sysadmin/grid5000/cassandra/Readme.md b/sysadmin/grid5000/cassandra/Readme.md
	index a13d76e..996b5c4 100644
	--- a/sysadmin/grid5000/cassandra/Readme.md
	+++ b/sysadmin/grid5000/cassandra/Readme.md
	@@ -1,257 +1,270 @@
	Grid5000 terraform provisioning
	===============================

	- [Grid5000 terraform provisioning](#grid5000-terraform-provisioning)
	- [Prerequisite](#prerequisite)
	- [Run](#run)
	- [Local (on vagrant)](#local-on-vagrant)
	- [On Grid5000](#on-grid5000)
	- [Via the custom script](#via-the-custom-script)
	+ - [Credentials](#credentials)
	- [Reservation configuration](#reservation-configuration)
	- [Nodes configuration](#nodes-configuration)
	- [Execution](#execution)
	- [(deprecated) With terraform](#deprecated-with-terraform)
	- [Cleanup](#cleanup)
	- [TODO](#todo)
	- [Possible improvments](#possible-improvments)

	Prerequisite
	------------

	Tools
	#####

	terraform >= 13.0
	vagrant >= 2.2.3 [for local tests only]

	Credentials
	###########

	* grid5000 credentials
	```
	cat <<EOF > ~/.grid5000.yml
	uri: https://api.grid5000.fr
	username: username
	password: password
	EOF
	```
	Theses credentials will be used to interact with the grid5000 api to create the jobs

	* Private/public key files (id_rsa) in the `~/.ssh` directory

	The public key will be installed on the nodes

	Run
	---

	### Local (on vagrant)

	The `Vagrantfile` is configured to provision 3 nodes, install cassandra and the configure the cluster using the ansible configuration:

	```
	vagrant up
	vagrant ssh cassandra1
	sudo -i
	nodetool status
	```

	If everything is ok, the `nodetool` command line returns:
	```
	root@cassandra1:~# nodetool status
	Datacenter: datacenter1
	=======================
	Status=Up/Down
	\|/ State=Normal/Leaving/Joining/Moving
	-- Address Load Tokens Owns (effective) Host ID Rack
	UN 10.168.180.12 15.78 KiB 256 67.9% 05d61a24-832a-4936-b0a5-39926f800d09 rack1
	UN 10.168.180.11 73.28 KiB 256 67.0% 23d855cc-37d6-43a7-886e-9446e7774f8d rack1
	UN 10.168.180.13 15.78 KiB 256 65.0% c6bc1eff-fa0d-4b67-bc53-fc31c6ced5bb rack1
	```

	Cassandra can take some time to start, so you have to wait before the cluster stabilize itself.

	### On Grid5000

	Useful link:
	Hardware information: https://www.grid5000.fr/w/Hardware
	Resources availability: https://www.grid5000.fr/w/Status

	#### Via the custom script

	+##### Credentials
	+
	+1. Create a file .vault_password with a password inside
	+2. execute the following commands :
	+```
	+ansible-vault create --vault-password-file=.vault_password .credentials
	+```
	+It will open an editor
	+3. Add the password of your journal user:
	+```
	+broker_consumer_password: <your password>
	+```
	##### Reservation configuration

	The configuration is defined on the `environment.cfg` file.
	In this file, g5k sites, cluster, nodes and reparition can be configured.

	##### Nodes configuration

	The node installation is done by ansible. It needs to know the node topology to correctly configure the tools (zfs pools and dataset, cassandra seed, ...)

	The configuration is centralized in the `ansible/hosts.yml` file

	##### Execution

	1. Transfer the files on g5k on the right site:

	```
	rsync -avP --exclude .vagrant --exclude .terraform cassandra access.grid5000.fr:/<site name>
	```

	2. Connect to the right site

	```
	ssh access.grid5000.fr
	ssh <site>
	```

	3. Reserve the disks

	The disks must be reserved before the node creation or they will not be detected on the nodes

	```
	./00-reserve_disks.sh
	```

	check the status of the job / the resources status to be sure they are correctly reserved

	```
	$ oarstat -fj <OAR JOB ID> \| grep state
	state = Running
	```
	The state must be running

	4. Launch a complete run

	For an interactive mode:
	```
	./01-run-interactive.sh
	```

	DISCLAIMER: Actually, it only runs the following steps:
	- reserve the nodes
	- install the os on all the nodes
	- launch ansible on all the nodes

	The underlying scripts can by run indepedently if they need to be restarted:
	- `02-reserver-nodes.sh`: Reserve the node resources
	- `03-deploy-nodes.sh`: Install the os (only one time per reservation) and launch ansible on all the nodes. To force an os resinstalltion, remove the `<JOB_ID_>.os.stamp` file


	In a reservation:

	TODO the scripts need to be reorganized to automatize the scheduled resource reservation

	1. Cleanup the resources

	To release the nodes:

	```
	oarstat -u
	<job id>
	<job id>
	```

	```
	oardel <jobid>
	```
	#### (deprecated) With terraform

	Terraform can be greate to reserve the resources but it doesn't not allow manage the scheduled jobs

	* Initialize terraform modules (first time only)
	```
	terraform init
	```

	* Test the plan

	It only check the status of the declared resources compared to the grid5000 status.
	It's a read only operation, no actions on grid5000 will be perform.

	```
	terraform plan
	```

	* Execute the plan

	```
	terraform apply
	```

	This action creates the job, provisions the nodes according the `main.tf` file content and install the specified linux distribution on it.

	This command will log the reserved node name in output.
	For example for a 1 node reservation:

	```
	grid5000_job.cassandra: Creating...
	grid5000_job.cassandra: Still creating... [10s elapsed]
	grid5000_job.cassandra: Creation complete after 11s [id=1814813]
	grid5000_deployment.my_deployment: Creating...
	grid5000_deployment.my_deployment: Still creating... [10s elapsed]
	grid5000_deployment.my_deployment: Still creating... [20s elapsed]
	grid5000_deployment.my_deployment: Still creating... [30s elapsed]
	grid5000_deployment.my_deployment: Still creating... [40s elapsed]
	grid5000_deployment.my_deployment: Still creating... [50s elapsed]
	grid5000_deployment.my_deployment: Still creating... [1m0s elapsed]
	grid5000_deployment.my_deployment: Still creating... [1m10s elapsed]
	grid5000_deployment.my_deployment: Still creating... [1m20s elapsed]
	grid5000_deployment.my_deployment: Still creating... [1m30s elapsed]
	grid5000_deployment.my_deployment: Still creating... [1m40s elapsed]
	grid5000_deployment.my_deployment: Still creating... [1m50s elapsed]
	grid5000_deployment.my_deployment: Still creating... [2m0s elapsed]
	grid5000_deployment.my_deployment: Still creating... [2m10s elapsed]
	grid5000_deployment.my_deployment: Creation complete after 2m12s [id=D-0bb76036-1512-429f-be99-620afa328b26]

	Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

	Outputs:

	nodes = [
	"chifflet-6.lille.grid5000.fr",
	]
	```
	It's now possible to connect to the nodes:
	```
	$ ssh -A access.grid5000.fr
	$ ssh -A root@chifflet-6.lille.grid5000.fr
	Linux chifflet-6.lille.grid5000.fr 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64
	Debian10-x64-base-2021060212
	(Image based on Debian Buster for AMD64/EM64T)
	Maintained by support-staff <support-staff@lists.grid5000.fr>

	Doc: https://www.grid5000.fr/w/Getting_Started#Deploying_nodes_with_Kadeploy

	root@chifflet-6:~#
	```

	Cleanup
	-------

	To destroy the resources before the end of the job:

	```
	terraform destroy
	```

	If the job is stopped, simply remove the `terraform.tfstate` file:
	```
	rm terraform.tfstate
	```

	## TODO

	[X] variablization of the script
	[X] Ansible provisionning of the nodes
	[X] disk initialization
	[X] support different cluster topologies (nodes / disks / ...)
	[X] cassandra installation
	[X] swh-storage installation
	[ ] journal client for mirroring
	[ ] monitoring by prometheus
	[ ] Add a tool to erase the reserved disks (useful to avoid zfs to detect the previous pools and be able to restart from scratch)

	## Possible improvments

	[ ] Use several besteffort jobs for cassandra nodes. They can be interrupted but don't have duration restrictions.
	[ ] Create a cassandra base image to speedup the environment initialization
	diff --git a/sysadmin/grid5000/cassandra/Vagrantfile b/sysadmin/grid5000/cassandra/Vagrantfile
	index 116dc98..4beb88e 100644
	--- a/sysadmin/grid5000/cassandra/Vagrantfile
	+++ b/sysadmin/grid5000/cassandra/Vagrantfile
	@@ -1,75 +1,82 @@
	# -- mode: ruby --
	# vi: set ft=ruby :

	vms = {
	"cassandra1" => {
	:ip => "10.168.180.11",
	:memory => 2048,
	:cpus => 2,
	:type => 'cassandra',
	},
	"cassandra2" => {
	:ip => "10.168.180.12",
	:memory => 2048,
	:cpus => 2,
	:type => 'cassandra',
	},
	"cassandra3" => {
	:ip => "10.168.180.13",
	:memory => 2048,
	:cpus => 2,
	:type => 'cassandra',
	},
	"swh-storage1" => {
	:ip => "10.168.180.14",
	:memory => 1024,
	:cpus => 2,
	:type => 'swh-storage',
	},
	}

	# Images/remote configuration
	$global_debian10_box = "debian10-20210517-1348"
	$global_debian10_box_url = "https://annex.softwareheritage.org/public/isos/libvirt/debian/swh-debian-10.9-amd64-20210517-1348.qcow2"

	vms.each { \| vm_name, vm_props \|

	Vagrant.configure("2") do \|global_config\|
	unless Vagrant.has_plugin?("libvirt")
	$stderr.puts <<-MSG
	vagrant-libvirt plugin is required for this.
	To install: `$ sudo apt install vagrant-libvirt
	MSG
	exit 1
	end

	global_config.vm.define vm_name do \|config\|
	config.vm.box = $global_debian10_box
	config.vm.box_url = $global_debian10_box_url
	config.vm.box_check_update = false
	config.vm.hostname = vm_name
	config.vm.network :private_network, ip: vm_props[:ip], netmask: "255.255.0.0"

	config.vm.synced_folder ".", "/vagrant", type: 'nfs', nfs_version: 4

	config.vm.provision :ansible do \|ansible\|
	ansible.verbose = true
	ansible.become = true
	ansible.playbook = "ansible/playbook.yml"
	ansible.inventory_path = "ansible/hosts.yml"
	+ ansible.raw_arguments = [
	+ "-v",
	+ "--connection=paramiko",
	+ "--private-key=/home/.../.vagrant/machines/.../private_key",
	+ "--extra-vars=@.credentials",
	+ "--vault-password-file=.vault_password"
	+ ]
	end

	config.vm.provider :libvirt do \|provider\|
	provider.memory = vm_props[:memory]
	provider.cpus = vm_props[:cpus]
	provider.driver = 'kvm'
	if vm_props[:type] == "cassandra"
	provider.storage :file, :size => '1G'
	provider.storage :file, :size => '1G'
	provider.storage :file, :size => '1G'
	end

	end
	end
	end
	}
	diff --git a/sysadmin/grid5000/cassandra/_provision_node.sh b/sysadmin/grid5000/cassandra/_provision_node.sh
	index fa438ae..a1234fe 100755
	--- a/sysadmin/grid5000/cassandra/_provision_node.sh
	+++ b/sysadmin/grid5000/cassandra/_provision_node.sh
	@@ -1,11 +1,11 @@
	#!/usr/bin/env bash

	set -eux

	apt update
	apt install -y ansible

	cd /root/install/ansible
	CASSANDRA_SEEDS="$(cat ../cassandra_seeds.lst)"

	-ansible-playbook -i hosts.yml -l "$(hostname)" playbook.yml --extra-vars "cassandra_seed_ips=${CASSANDRA_SEEDS}"
	+ansible-playbook -i hosts.yml -e @/root/install/.credentials --vault-password-file=/root/install/.vault_password -l "$(hostname)" playbook.yml --extra-vars "cassandra_seed_ips=${CASSANDRA_SEEDS}"
	diff --git a/sysadmin/grid5000/cassandra/ansible/_configure_replayer_services.yml b/sysadmin/grid5000/cassandra/ansible/_configure_replayer_services.yml
	new file mode 100644
	index 0000000..11a5dee
	--- /dev/null
	+++ b/sysadmin/grid5000/cassandra/ansible/_configure_replayer_services.yml
	@@ -0,0 +1,23 @@
	+---
	+- name: swh replayer {{obj}} configuration files
	+ template:
	+ src: templates/swhstorage/replayer.yml
	+ dest: /etc/softwareheritage/replayer/replayer-{{ obj }}.yml
	+
	+- name: swh replayer replayer-{{obj}}@
	+ template:
	+ src: templates/swhstorage/replayer@.service
	+ dest: /etc/systemd/system/replayer-{{obj}}@.service
	+
	+- name: swh replayer {{obj}} target
	+ template:
	+ src: templates/swhstorage/replayer.target
	+ dest: /etc/systemd/system/replayer-{{obj}}.target
	+ vars:
	+ process_count: "{{ hostvars[inventory_hostname]['journal']['replayer_count'][obj] }}"
	+
	+- name: start {{obj}} replayer target
	+ service:
	+ name: replayer-{{obj}}.target
	+ enabled: true
	+ state: started
	diff --git a/sysadmin/grid5000/cassandra/ansible/cassandra.yml b/sysadmin/grid5000/cassandra/ansible/cassandra.yml
	index 4fd44bb..b2c828f 100644
	--- a/sysadmin/grid5000/cassandra/ansible/cassandra.yml
	+++ b/sysadmin/grid5000/cassandra/ansible/cassandra.yml
	@@ -1,58 +1,59 @@
	---
	# - name: "Get public ipv4 address"
	# set_fact:
	# cassandra_seed_ips: "{{ansible_facts[item]['ipv4']['address']}}"
	# with_items:
	# - "{{cassandra_listen_interface }}"

	- name: Install cassandra signing key
	apt_key:
	url: https://downloads.apache.org/cassandra/KEYS
	state: present

	- name: Install cassandra apt repository
	apt_repository:
	repo: deb http://downloads.apache.org/cassandra/debian 40x main
	state: present
	filename: cassandra.sources

	- name: Install cassandra packages
	apt:
	update_cache: true # force an apt update before
	name:
	## TODO: check other version than jdk11
	- openjdk-11-jdk
	- cassandra
	- dstat
	- smartmontools
	- facter
	+ - tcpdump

	- name: Create datadirs
	file:
	state: directory
	path: "{{ item }}"
	owner: "cassandra"
	group: "cassandra"
	mode: "0755"
	with_items:
	- "{{ cassandra_data_dir_base }}"
	- "{{ cassandra_data_dir_system }}"
	- "{{ cassandra_data_dir }}"
	- "{{ cassandra_commitlogs_dir }}"

	- name: Configure cassandra
	template:
	src: "templates/{{item}}"
	dest: "{{cassandra_config_dir}}/{{item}}"
	with_items: [cassandra.yaml, jvm.options]
	register: cassandra_configuration_files

	- name: Restart cassandra service
	service:
	name: cassandra
	state: restarted
	when: cassandra_configuration_files.changed



	# TODO test different read ahead
	diff --git a/sysadmin/grid5000/cassandra/ansible/hosts.yml b/sysadmin/grid5000/cassandra/ansible/hosts.yml
	index a530bda..7845051 100644
	--- a/sysadmin/grid5000/cassandra/ansible/hosts.yml
	+++ b/sysadmin/grid5000/cassandra/ansible/hosts.yml
	@@ -1,88 +1,126 @@
	# Global configuration
	swh-storage:
	hosts:
	parasilo-[20:28].rennes.grid5000.fr:
	+ paranoia-[1:8].rennes.grid5000.fr:
	# local vagrant hosts
	swh-storage1:
	vars:
	ansible_connection: local
	+ journal:
	+ brokers:
	+ - broker1.journal.softwareheritage.org:9093
	+ - broker2.journal.softwareheritage.org:9093
	+ - broker3.journal.softwareheritage.org:9093
	+ - broker4.journal.softwareheritage.org:9093
	+ consumer:
	+ user: swh-vse
	+ group: swh-vse-grid5000-1
	+ replayer_count:
	+ content: 0
	+ skipped_content: 0
	+ directory: 50
	+ origin: 0
	+ origin_visit: 0
	+ origin_visit_status: 0
	+ release: 0
	+ revision: 0
	+ snapshot: 0
	+

	cassandra:
	hosts:
	dahu-[1:32].grenoble.grid5000.fr:
	parasilo-[1:19].rennes.grid5000.fr:
	# local vagrant hosts
	cassandra[1:9]:
	vars:
	ansible_connection: local
	cassandra_config_dir: /etc/cassandra
	cassandra_data_dir_base: /srv/cassandra
	cassandra_data_dir_system: "{{cassandra_data_dir_base}}/system"
	cassandra_data_dir: "{{ cassandra_data_dir_base }}/data"
	cassandra_commitlogs_dir: "{{ cassandra_data_dir_base }}/commitlogs"


	# Per cluster specificities
	dahu_cluster_hosts:
	hosts:
	dahu[1:32].grenoble.grid5000.fr
	vars:
	cassandra_listen_interface: enp24s0f0

	parasilo_cluster_hosts:
	hosts:
	parasilo-[1:28].rennes.grid5000.fr:
	+ paranoia-[1:8].rennes.grid5000.fr:
	vars:
	cassandra_listen_interface: eno1
	zfs_pools:
	commitlogs:
	disks:
	- sdf
	datasets:
	commitlogs: /srv/cassandra/commitlogs
	data:
	disks:
	- sdb
	- sdc
	- sdd
	- sde
	datasets:
	data: /srv/cassandra/data

	-
	# Vagrant configuration
	vagrant_nodes:
	hosts:
	cassandra1:
	ansible_host: 10.168.180.11
	ansible_user: vagrant
	ansible_ssh_private_key_file: .vagrant/machines/cassandra1/libvirt/private_key
	cassandra2:
	ansible_host: 10.168.180.12
	ansible_user: vagrant
	ansible_ssh_private_key_file: .vagrant/machines/cassandra2/libvirt/private_key
	cassandra3:
	ansible_host: 10.168.180.13
	ansible_user: vagrant
	ansible_ssh_private_key_file: .vagrant/machines/cassandra3/libvirt/private_key
	swh-storage1:
	ansible_host: 10.168.180.14
	ansible_user: vagrant
	ansible_ssh_private_key_file: .vagrant/machines/swh-storage/libvirt/private_key
	vars:
	ansible_connection: ssh
	+ journal:
	+ brokers:
	+ # staging
	+ - broker0.journal.staging.swh.network:9093
	+ consumer:
	+ user: swh-vse
	+ group: swh-vse-grid5000-1
	+ replayer_count:
	+ content: 0
	+ skipped_content: 0
	+ directory: 5
	+ origin: 0
	+ origin_visit: 0
	+ origin_visit_status: 0
	+ release: 0
	+ revision: 0
	+ snapshot: 0
	cassandra_listen_interface: eth1
	# passed through --extra-vars on grid5000
	cassandra_seed_ips: 10.168.180.11,10.168.180.12,10.168.180.13
	zfs_pools:
	commitlogs:
	disks:
	- vdb
	datasets:
	commitlogs: /srv/cassandra/commitlogs
	data:
	disks:
	- vdc
	- vdd
	datasets:
	data: /srv/cassandra/data
	diff --git a/sysadmin/grid5000/cassandra/ansible/swh-storage.yml b/sysadmin/grid5000/cassandra/ansible/swh-storage.yml
	index 2ae0901..e270fc1 100644
	--- a/sysadmin/grid5000/cassandra/ansible/swh-storage.yml
	+++ b/sysadmin/grid5000/cassandra/ansible/swh-storage.yml
	@@ -1,101 +1,120 @@
	---
	- name: Add Backports repository
	apt_repository:
	repo: deb http://deb.debian.org/debian/ buster-backports main contrib non-free
	filename: backports.sources

	- name: swhstorage group
	group:
	name: swhstorage

	- name: swhstorage user
	user:
	name: swhstorage
	group: swhstorage

	- name: Add SWH repository
	apt_repository:
	repo: deb [trusted=yes] https://debian.softwareheritage.org/ buster-swh main
	state: present
	filename: cassandra.sources

	- name: Install packages
	apt:
	name:
	+ - daemonize
	- dstat
	+ - facter
	- python3
	- python3-gunicorn
	+ - tcpdump

	- name: Install packages from backports
	apt:
	name:
	- python3-typing-extensions
	- gunicorn3
	default_release: buster-backports

	- name: Install swh storage packages
	apt:
	name:
	- python3-swh.storage
	+ - python3-swh.journal

	- name: Create directories
	file:
	state: directory
	path: "{{ item }}"
	owner: root
	group: root
	mode: "0755"
	with_items:
	- /etc/gunicorn
	- /etc/gunicorn/instances
	- /run/gunicorn
	- /run/gunicorn/swh-storage
	- /etc/softwareheritage
	- /etc/softwareheritage/storage
	+ - /etc/softwareheritage/replayer

	- name: Create swh-storage directories
	file:
	state: directory
	path: "{{ item }}"
	owner: swhstorage
	group: swhstorage
	mode: "0755"
	with_items:
	- /run/gunicorn/swh-storage/
	+ - /run/replayer

	- name: Configure gunicorn - default service
	template:
	src: "templates/gunicorn/gunicorn.service"
	dest: "/etc/systemd/system/gunicorn.service"

	- name: Configure gunicorn - log configuration
	template:
	src: "templates/gunicorn/logconfig.ini"
	dest: "/etc/gunicorn/logconfig.ini"

	- name: swh-storage gunicorn instance configuration
	template:
	src: "templates/gunicorn/gunicorn-instance.cfg"
	dest: "/etc/gunicorn/instances/swh-storage.cfg"

	- name: swh-storage configuration directories
	template:
	src: "templates/swhstorage/storage.yml"
	dest: "/etc/softwareheritage/storage/storage.yml"

	- name: swh-storage service configuration
	template:
	src: "templates/gunicorn/gunicorn-instance-service.cfg"
	dest: "/etc/systemd/system/gunicorn-swh-storage.service" # TODO variabilize

	- name: swh-storage service
	service:
	name: gunicorn-swh-storage
	enabled: true
	state: started

	- name: swh-storage init cassandra script
	template:
	src: templates/swhstorage/init-cassandra.sh
	dest: /usr/local/bin/swh-storage-init-cassandra.sh
	mode: 0755
	+
	+- name: Configure replayer services
	+ include: _configure_replayer_services.yml obj={{ item }}
	+ loop:
	+ - content
	+ - skipped_content
	+ - directory
	+ - origin
	+ - origin_visit
	+ - origin_visit_status
	+ - release
	+ - revision
	+ - snapshot
	diff --git a/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer.target b/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer.target
	new file mode 100644
	index 0000000..9ab3348
	--- /dev/null
	+++ b/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer.target
	@@ -0,0 +1,7 @@
	+[Unit]
	+Description=swh storage replayer {{ item }}
	+Requires={% for idx in range(process_count \| int) %}replayer-{{ item }}@{{idx + 1}}.service {% endfor %}
	+
	+
	+[Install]
	+WantedBy=multi-user.target
	diff --git a/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer.yml b/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer.yml
	new file mode 100644
	index 0000000..fa42f3b
	--- /dev/null
	+++ b/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer.yml
	@@ -0,0 +1,23 @@
	+storage:
	+ cls: cassandra
	+ args:
	+ hosts: {{ cassandra_seed_ips.split(',') }}
	+ keyspace: swh
	+ objstorage:
	+ cls: memory
	+ args: {}
	+
	+journal_client:
	+ cls: kafka
	+ brokers:
	+{% for broker in journal.brokers %}
	+ - {{ broker }}
	+{% endfor %}
	+ sasl.username: {{ journal.consumer.user }}
	+ sasl.password: {{ journal_consumer_password }}
	+ security.protocol: sasl_ssl
	+ sasl.mechanism: SCRAM-SHA-512
	+ # The prefix must match the username
	+ group_id: {{ journal.consumer.group }}
	+ object_types:
	+ - {{ item }}
	diff --git a/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer@.service b/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer@.service
	new file mode 100644
	index 0000000..05989c9
	--- /dev/null
	+++ b/sysadmin/grid5000/cassandra/ansible/templates/swhstorage/replayer@.service
	@@ -0,0 +1,20 @@
	+[Unit]
	+Description=swh storage {{ item }} replayer
	+ConditionPathExists=/etc/softwareheritage/replayer/replayer-{{ item }}.yml
	+PartOf=replayer-{{item}}.target
	+
	+[Service]
	+Type=forking
	+User=swhstorage
	+Group=swhstorage
	+PIDFile=/run/replayer/replayer-{{item}}-%i.pid
	+Environment=SWH_CONFIG_FILENAME=/etc/softwareheritage/replayer/replayer-{{ item }}.yml
	+Environment=SWH_LOG_TARGET=journal
	+ExecStart=daemonize -p /run/replayer/replayer-{{item}}-%i.pid /usr/bin/swh storage replay
	+ExecStop=/bin/kill -TERM $MAINPID
	+ExecReload=/bin/kill -HUP $MAINPID
	+# Restart=Always
	+# RestartSec=10
	+
	+[Install]
	+WantedBy=multi-user.target

File Metadata

Mime Type: text/x-diff
Expires: Fri, Jul 4, 11:00 AM (3 w, 6 d ago)
Storage Engine: blob
Storage Format: Raw Data
Storage Handle: 3347948

No OneTemporaryActions

View Options

File Metadata

Event Timeline

No OneTemporary
Actions