- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Aug 19 2022
Aug 19 2022
fix init database command
Add an init cassandra option
remove port from cassandra host
Improve indentation v4
Improve indentation v3
Improve indentation
improve indentation
fix the configmap name
vsellier committed R260:eb40b1cbbc6c: add more default values for cassandra replayer (authored by vsellier).
add more default values for cassandra replayer
vsellier committed rSKCONF6a8eab9de494: Configure the cassandra-production cluster (authored by vsellier).
Configure the cassandra-production cluster
vsellier committed rSKCONF438c33c127f1: Declare the swh namespace for the production-cassandra cluster (authored by vsellier).
Declare the swh namespace for the production-cassandra cluster
specify the journal client user
prepare production deployment
vsellier committed rSPSITE5d845aa0805a: cassandra: extend directory permission to allow monitoring to check the disks (authored by vsellier).
cassandra: extend directory permission to allow monitoring to check the disks
Aug 18 2022
Aug 18 2022
all server reconfigured and cassandra started on them:
/opt/cassandra/bin/nodetool status Datacenter: sesi_rocquencourt ============================= Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 192.168.100.184 88.65 KiB 16 34.3% e0c24d24-6f68-4a26-8561-94e67b58211a rack1 UN 192.168.100.181 84.71 KiB 16 31.3% 1d9b9e7d-b376-4afe-8f67-482e8412f21b rack1 UN 192.168.100.186 69.07 KiB 16 34.2% 0dd3426d-9159-47bd-9b4e-065ff0fbb889 rack1 UN 192.168.100.183 69.08 KiB 16 37.1% 78281a92-7fa0-43bd-bc33-c5b419ee8715 rack1 UN 192.168.100.185 69.07 KiB 16 32.2% abf9b69e-3cec-4ac3-a195-a54481e4d9da rack1 UN 192.168.100.182 74.05 KiB 16 30.9% eca5ea5d-8bd5-4301-9a5e-ffa01aa1b7e5 rack1
vsellier closed T4446: Prepare the disks and configure zfs, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
Recreating the zpool correctly:
# mixedused ls /dev/disk/by-id/nvme-MO003200KXAVU* | grep -v part | xargs -t zpool create -o ashift=12 -O mountpoint=none mixeduse zfs create -o mountpoint=/srv/cassandra/instance1/data mixeduse/cassandra-instance1-data
Testing the performances of the different configuration (on a zfs pool with only one disk):
- disk block: 512k / zpool ashift:9
zpool create -o ashift=9 -O mountpoint=none mixeduse /dev/disk/by-id/nvme-MO003200KXAVU_SJA4N7938I0405A0U
zfs create -o mountpoint=/srv/cassandra/instance1/data -o atime=off -o relatime=on mixeduse/cassandra-data
cd /srv/cassandra/instance1/data
bonnie++ -d . -m cassandra04 -u nobody
Using uid:65534, gid:65534.
Writing a byte at a time...done
Writing intelligently...done
Rewriting...done
Reading a byte at a time...done
Reading intelligently...done
start 'em...done...done...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Version 2.00 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
cassandra04 515496M 293k 99 1.0g 99 703m 99 661k 99 1.4g 91 13717 463
Latency 48216us 7316us 8224us 23303us 7928us 1606us
Version 2.00 ------Sequential Create------ --------Random Create--------
cassandra04 -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 16384 98 +++++ +++ 16384 8 +++++ +++ +++++ +++ 16384 99
Latency 2679us 1207us 4851ms 2850us 138us 301us
1.98,2.00,cassandra04,1,1659338044,515496M,,8192,5,293,99,1080974,99,720299,99,661,99,1488832,91,13717,463,16,,,,,28232,98,+++++,+++,2018,8,+++++,+++,+++++,+++,24821,99,48216us,7316us,8224us,23303us,7928us,1606us,2679us,1207us,4851ms,2850us,138us,301usThe nvme format command didn't succeed on the write intensive disk. It never exits and the disk become unresponsive after that.
vsellier changed the status of T4446: Prepare the disks and configure zfs from Open to Work in Progress.
cassandra: Add missing dependencies
vsellier committed rSPSITEa6e31c14466c: cassandra: Remove the no longer used cassandra.yaml template (authored by vsellier).
cassandra: Remove the no longer used cassandra.yaml template
vsellier committed rSPRE8791a2ec762d: cassandra: Cleanup extra mounts of the cassandra directories (authored by vsellier).
cassandra: Cleanup extra mounts of the cassandra directories
vsellier committed rSPSITE57d978d9e3de: cassandra: Refactor the former installation scripts (authored by vsellier).
cassandra: Refactor the former installation scripts
Aug 17 2022
Aug 17 2022
vsellier committed R260:8f12e7df4312: Bootstrap a swh chart only with the cassandra replayer to starting (authored by vsellier).
Bootstrap a swh chart only with the cassandra replayer to starting
add swh-storage-replayer application
use inline_yaml instead of to_yaml
thanks for the hint, I completely forgot about it
Aug 16 2022
Aug 16 2022
fix the private data repository url
fix indentation
fix indentation
vsellier committed rSKCONF6f65eb6f12f0: Configure the argocd applications configuration (authored by vsellier).
Configure the argocd applications configuration
Activate applications autosync
vsellier committed rSKCONF59bd7205efc8: declare argocd own applications and prepare the cassandra cluster ones (authored by vsellier).
declare argocd own applications and prepare the cassandra cluster ones
vsellier committed R261:15987cb50831: Declare the private repository that will contain the k8s clusters secrets (authored by vsellier).
Declare the private repository that will contain the k8s clusters secrets
- Rebase
- Override the jmx port value via the JVM_EXTRA_OPTS environment
- Inline the cassandra.yaml properties in hiera
Increase cassandra nodes memory
Aug 12 2022
Aug 12 2022
Setting -Dcassandra.jmx.local.port / -Dcassandra.jmx.remote.port / -Dcom.sun.management.jmxremote.rmi.port in the $JVM_EXTRA_OPTS env var should override what this file configured
I specified why on the first line of the file, but I realize it not very visible.
vsellier renamed T4435: scrubber log verbosity puts a risk on ELK from scrubber log verbosity put a risk on ELK to scrubber log verbosity puts a risk on ELK .
vsellier changed the status of T4391: [cassandra] deploy the replaying stack, a subtask of T4373: [cassandra] Test the new hardware, from Open to Work in Progress.
vsellier changed the status of T4391: [cassandra] deploy the replaying stack from Open to Work in Progress.
The puppet code is ready for review. It was updated to support multi instances deployment in anticipation of T4375.
Aug 11 2022
Aug 11 2022
vsellier committed rSPSITEfbbc0eb13e49: vagrant: fix the internal_network declaration (authored by vsellier).
vagrant: fix the internal_network declaration
vsellier committed rSENVaef11d699700: vagrant: Fix the initial environment synchronization (authored by vsellier).
vagrant: Fix the initial environment synchronization
Aug 10 2022
Aug 10 2022
vsellier committed rSPRE5fc8dc9b2626: cassandra: expose the log directory to the pods (authored by vsellier).
cassandra: expose the log directory to the pods
For the record, the issues related to the commitlog_directory configuration:
After spending some time to successfully start a cassandra cluster of 2 nodes with a declarative configuration), these are the observations:
- A service can't be used to expose the cassandra ports to the clustrer, the pod address must be used. It's because cassandra use the dns name provided as listen address
- It should work by setting the listen address to 0.0.0.0 but it's stongly recommanded to not use this in the documentation
Setting listen_address to 0.0.0.0 is always wrong.
- Using internal pod address will avoid multi DC deployment for the future
vsellier committed rSKCONF8e37b0f5df0c: cassandra: poc working manual declarative configuration (authored by vsellier).
cassandra: poc working manual declarative configuration
Aug 9 2022
Aug 9 2022
vsellier committed rSPRE73d451f1b506: cassandra: Remove ballooning for the control-plane nodes (authored by vsellier).
cassandra: Remove ballooning for the control-plane nodes
vsellier committed R260:6908e14f5cf3: workers: try to stop argocd to override the keda status (authored by vsellier).
workers: try to stop argocd to override the keda status
Aug 8 2022
Aug 8 2022
VPN certificate revoked in opensense
vsellier changed the status of T4414: Outboard sysadm intern Amadou Thiam from Open to Work in Progress.
vsellier accepted D8212: docker: add a generic 'swh' command in storage and worker entrypoint scripts.
looks good, thanks.
I'm not sure if it's not better to keep the KAFKA_BROKER_ID closed to the instance declaration as it needs to be unique in a cluster and can't be shared between the instances.
But otherwise, thank, it's simpler like that.
Jul 13 2022
Jul 13 2022
vsellier committed rSKCONFff5707a7c44b: Bootstrap manual cassandra configuration (authored by vsellier).
Bootstrap manual cassandra configuration
non working k8ssandra configuration
bootstrap the cassandra cluster
vsellier committed rSPRE944ad7507847: cassandra: update the cluster configuration to deploy cassadra manually (authored by vsellier).
cassandra: update the cluster configuration to deploy cassadra manually
vsellier committed rSPRE16feb3dda32c: Deploy the cassandra operator on the production cassandra cluster (authored by vsellier).
Deploy the cassandra operator on the production cassandra cluster
Unfortunately, the operator test is a failure due to the lack of configuration possibility
- non blocker, the init containers are OOMkilled during the start, it can be solved by editing the cassandra statefulset created by the operator to extend the limits
- blocker, it's not possible to configure the commitlog_directory explicitly. it's by default on /var/lib/cassandra/commitlog
- it's not easy to propagate the host mounts to use 2 mountpoints /srv/cassandra and /srv/cassandra/commitlog without tweaking the kernel / rancher configuration
- it's not possible to add a second volume on the pod description created by the operator
Jul 12 2022
Jul 12 2022
vsellier renamed T4391: [cassandra] deploy the replaying stack from [cassandra] deploy the replying stack to [cassandra] deploy the replaying stack.
vsellier updated the diff for D8116: Deploy the cassandra operator on the production cassandra cluster.
declare the extra volume binding (no impact on the tfstate)
vsellier requested review of D8116: Deploy the cassandra operator on the production cassandra cluster.
cassandra - match node roles
cassandra - match kubernes version
vsellier committed rSKCONFf89d3a7a6bc4: Add the argo workflow poc configuration (authored by vsellier).
Add the argo workflow poc configuration
vsellier changed the status of T4374: [cassandra] Test basic topology, a subtask of T4379: [cassandra] create etcd / controlplane servers, from Open to Work in Progress.
vsellier changed the status of T4374: [cassandra] Test basic topology from Open to Work in Progress.
The mountpoint needs to be declare on the kubelet container to be reachable by the pods:
--- /tmp/cluster-orig.yaml 2022-07-12 11:27:27.169509573 +0200 +++ /tmp/cluster.yaml 2022-07-12 11:26:54.865395186 +0200 @@ -58,6 +58,8 @@ service_node_port_range: 30000-32767 kube-controller: {} kubelet: + extra_binds: + - '/srv/prometheus:/srv/prometheus' fail_swap_on: false generate_serving_certificate: false kubeproxy: {}
vsellier closed T4389: [cassandra] Configure the monitoring of the cluster, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
vsellier committed rSKCONF18f544855355: Bootstrap cassandra cluster configuration (authored by vsellier).
Bootstrap cassandra cluster configuration
Specify the repository organization
vsellier committed R261:8249ffa761b1: Add the kubernetes clusters configuration repository (authored by vsellier).
Add the kubernetes clusters configuration repository
Jul 11 2022
Jul 11 2022
vsellier committed rSPREbd8e7bc4223d: Declare the kubernetes cluster and management nodes for cassandra (authored by vsellier).
Declare the kubernetes cluster and management nodes for cassandra
vsellier updated the diff for D8094: Declare the kubernetes cluster and management nodes for cassandra.
add tfstate file
Configure the data directory:
root@pergamon:~# clush -b -w @cassandra-mgmt hostname --------------- rancher-node-cassandra1 --------------- rancher-node-cassandra1 --------------- rancher-node-cassandra2 --------------- rancher-node-cassandra2 --------------- rancher-node-cassandra3 --------------- rancher-node-cassandra3
vsellier renamed T4389: [cassandra] Configure the monitoring of the cluster from Configure the monitoring of the cluster to [cassandra] Configure the monitoring of the cluster.
vsellier moved T4389: [cassandra] Configure the monitoring of the cluster from Backlog to in-progress on the System administration board.
vsellier changed the status of T4389: [cassandra] Configure the monitoring of the cluster from Open to Work in Progress.
Finally, the cluster is up.
I'm not sure what unstuck the node registration, but I suspect a node with all the roles is needed to bootstrap the cluster.
I tried this initially, it didn't worked, but I'm not sure in which status the cluster was.
vsellier closed T4379: [cassandra] create etcd / controlplane servers, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
Force to use pergamon as dns
update facts