- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Feed Advanced Search
Advanced Search
Advanced Search
Sep 20 2022
Sep 20 2022
vsellier changed the status of T4544: cgit loader: the repository detail is not necessary the summary tab from Open to Work in Progress.
Sep 19 2022
Sep 19 2022
vsellier committed R260:b4f132e09f05: add-forge-now: declare loaders for staging and production environments (authored by vsellier).
add-forge-now: declare loaders for staging and production environments
vsellier updated the diff for D8507: add-forge-now: declare loaders for staging and production environments.
Reduce queueThreshold to ensure running at max loaders when needed
vsellier requested review of D8507: add-forge-now: declare loaders for staging and production environments.
vsellier changed the status of T4541: Deploy git loaders dedicated to load add forge now requests from Open to Work in Progress.
vsellier committed rSKCONF368dd337fec7: Configure applications using swh-charts repository to use a dedicated… (authored by vsellier).
Configure applications using swh-charts repository to use a dedicated…
vsellier committed rSKCONF9ddeb0be7270: cassadra-replay: rename the argocd application to be able to deploy an swh stack (authored by vsellier).
cassadra-replay: rename the argocd application to be able to deploy an swh stack
vsellier committed rSKCONFea89f7842be4: cassandra-replay: use the master branch to deploy cassandra replayers (authored by vsellier).
cassandra-replay: use the master branch to deploy cassandra replayers
vsellier committed R260:904535431a5a: cassandra-replay: Configure the replayer environment (authored by vsellier).
cassandra-replay: Configure the replayer environment
vsellier committed rSKCONFec5eee38a731: cassandra: Add a panel showing the un-repaired size per table (authored by vsellier).
cassandra: Add a panel showing the un-repaired size per table
vsellier committed rSPSITEc7ec39553258: postfix: remove zack from the root smtp alias (authored by vsellier).
postfix: remove zack from the root smtp alias
update commit message
Sep 18 2022
Sep 18 2022
vsellier committed R260:f9efd6271e6f: Redispatch replayers to prioritize origin-visit-status topic (authored by vsellier).
Redispatch replayers to prioritize origin-visit-status topic
Merge branch 'master' into cassandra
Sep 17 2022
Sep 17 2022
vsellier committed rSKCONF4c74f4d7a79f: production: fix reaper ServiceMonitor declaration (authored by vsellier).
production: fix reaper ServiceMonitor declaration
Sep 16 2022
Sep 16 2022
production: add reaper monitoring
vsellier committed rSKCONFaf25578e8cca: Deploy reaper on production-cluster to manager repair jobs (authored by vsellier).
Deploy reaper on production-cluster to manager repair jobs
For example, thes logs were saw in the reaper logs when several repairs for different keystores are scheduler at the same time:
│ INFO [2022-09-16 15:02:34,268] [archive_production:67b1d310-35c5-11ed-8ea7-4b43418aeab2:67b9e963-35c5-11ed-8ea7-4b43418aeab2] i.c.s.SegmentRunner - Repair for segment 67b9e963-35c5-11ed-8ea7-4b43418aeab2 started, status wait will timeout in 1800000 millis │ │ INFO [2022-09-16 15:02:58,602] [archive_production:9a773740-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Maximum number of concurrent repairs reached. Repair 9a773740-35cf-11ed-8ea7-4b43418aeab2 will resume later. │ │ INFO [2022-09-16 15:02:58,602] [archive_production:9a773740-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Current active repair runners: [(67b1d310-35c5-11ed-8ea7-4b43418aeab2,1663335748289), (76982be0-35cf-11ed-8ea7-4b43418aeab2,1663340068254), (9a773740-35cf-11ed-8ea7-4b43418aeab2,1663340128436), (9a9cc0a0-35cf-1 │ │ INFO [2022-09-16 15:02:58,787] [archive_production:9a9cc0a0-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Maximum number of concurrent repairs reached. Repair 9a9cc0a0-35cf-11ed-8ea7-4b43418aeab2 will resume later. │ │ INFO [2022-09-16 15:02:58,787] [archive_production:9a9cc0a0-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Current active repair runners: [(67b1d310-35c5-11ed-8ea7-4b43418aeab2,1663335748289), (76982be0-35cf-11ed-8ea7-4b43418aeab2,1663340068254), (9a773740-35cf-11ed-8ea7-4b43418aeab2,1663340128436), (9a9cc0a0-35cf-1 │ │ INFO [2022-09-16 15:02:59,336] [archive_production:9aeae0a0-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Maximum number of concurrent repairs reached. Repair 9aeae0a0-35cf-11ed-8ea7-4b43418aeab2 will resume later. │ │ INFO [2022-09-16 15:02:59,336] [archive_production:9aeae0a0-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Current active repair runners: [(67b1d310-35c5-11ed-8ea7-4b43418aeab2,1663335748289), (76982be0-35cf-11ed-8ea7-4b43418aeab2,1663340068254), (9a773740-35cf-11ed-8ea7-4b43418aeab2,1663340128436), (9a9cc0a0-35cf-1 │ │ INFO [2022-09-16 15:02:59,555] [archive_production:9b0c7260-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Maximum number of concurrent repairs reached. Repair 9b0c7260-35cf-11ed-8ea7-4b43418aeab2 will resume later. │ │ INFO [2022-09-16 15:02:59,555] [archive_production:9b0c7260-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Current active repair runners: [(67b1d310-35c5-11ed-8ea7-4b43418aeab2,1663335748289), (76982be0-35cf-11ed-8ea7-4b43418aeab2,1663340068254), (9a773740-35cf-11ed-8ea7-4b43418aeab2,1663340128436), (9a9cc0a0-35cf-1 │ │ INFO [2022-09-16 15:02:59,779] [archive_production:76982be0-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Attempting to run new segment... │ │ INFO [2022-09-16 15:02:59,813] [archive_production:76982be0-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Next segment to run : 76998b71-35cf-11ed-8ea7-4b43418aeab2 │ │ INFO [2022-09-16 15:02:59,849] [archive_production:76982be0-35cf-11ed-8ea7-4b43418aeab2:76998b71-35cf-11ed-8ea7-4b43418aeab2] i.c.j.JmxProxy - Triggering repair of range (-5797115047693728403,-5671075333212739092] for keyspace "reaper_db" on host 192.168.100.182, with repair parallelism dc_parallel, in cluster with Cas │ │ INFO [2022-09-16 15:02:59,851] [archive_production:76982be0-35cf-11ed-8ea7-4b43418aeab2:76998b71-35cf-11ed-8ea7-4b43418aeab2] i.c.j.JmxProxy - Triggering repair for ranges -5797115047693728403:-5671075333212739092 │ │ INFO [2022-09-16 15:02:59,863] [archive_production:76982be0-35cf-11ed-8ea7-4b43418aeab2:76998b71-35cf-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Triggered repair of segment 76998b71-35cf-11ed-8ea7-4b43418aeab2 via host 192.168.100.182 │ │ INFO [2022-09-16 15:02:59,863] [archive_production:76982be0-35cf-11ed-8ea7-4b43418aeab2:76998b71-35cf-11ed-8ea7-4b43418aeab2] i.c.s.SegmentRunner - Repair for segment 76998b71-35cf-11ed-8ea7-4b43418aeab2 started, status wait will timeout in 1800000 millis │ │ INFO [2022-09-16 15:03:04,227] [archive_production:67b1d310-35c5-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - Attempting to run new segment... │ │ INFO [2022-09-16 15:03:04,254] [archive_production:67b1d310-35c5-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - All nodes are busy or have too many pending compactions for the remaining candidate segments. │ │ INFO [2022-09-16 15:03:04,262] [archive_production:67b1d310-35c5-11ed-8ea7-4b43418aeab2] i.c.s.RepairRunner - All nodes are busy or have too many pending compactions for the remaining candidate segments.
perhaps a little concern regarding the length of the group_id but nothing blocking
Reaper was manually deployed and running.
The main functionnalities for now are the scheduling of the different repair type, the orchestration of the segment to repair to avoid a repair of the same segment in different replicas.
Secondary functionalities can be useful too like the repair progress, stop / resume http://cassandra-reaper.io/docs/concepts/
Sep 15 2022
Sep 15 2022
The group id of the authenticated consumers have to be probably updated to match the kafka acls
vsellier committed rSPSITE386b3c6a40e3: rancher: use a zfs dataset for /var/lib/kubelet to use the local storage for… (authored by vsellier).
rancher: use a zfs dataset for /var/lib/kubelet to use the local storage for…
vsellier updated the diff for D8482: rancher: use a zfs dataset for /var/lib/kubelet to use the local storage for EmptyDir volumes.
rebase
vsellier closed T4506: Use local hypervisor storage in the loader pods, a subtask of T4144: Elastic worker infrastructure, as Resolved.
Example during the loading of https://github.com/torvalds/linux by a pod:
% /usr/sbin/zfs list data/docker data/kubelet NAME USED AVAIL REFER MOUNTPOINT data/docker 3.81G 40.4G 83.2M /var/lib/docker data/kubelet 3.71G 40.4G 3.71G /var/lib/kubelet
The compression is not as useful as for docker
% /usr/sbin/zfs get compressratio data/kubelet data/docker NAME PROPERTY VALUE SOURCE data/docker compressratio 2.95x - data/kubelet compressratio 1.07x -
vsellier committed R260:3f0e96382e4d: loaders: Use an emptydir volume type for /tmp (authored by vsellier).
loaders: Use an emptydir volume type for /tmp
vsellier updated the diff for D8482: rancher: use a zfs dataset for /var/lib/kubelet to use the local storage for EmptyDir volumes.
rebase
vsellier committed rSPPRIVC834df81ebe3a: refresh censored data compared to the private data (authored by vsellier).
refresh censored data compared to the private data
vsellier committed rSPSITEde429d5dfd80: cassandra: Allow to configure the jmx for remote or local only access (authored by vsellier).
cassandra: Allow to configure the jmx for remote or local only access
vsellier added a comment to D8482: rancher: use a zfs dataset for /var/lib/kubelet to use the local storage for EmptyDir volumes.
heh sorry for the title mess
vsellier committed rSPRE444d013f59ef: staging-cluster: Declare the 2d disk on the local storage of uffizi (authored by vsellier).
staging-cluster: Declare the 2d disk on the local storage of uffizi
The kubelet dataset will need to be manually created on all the rancher nodes (except staging worker2 and worker3 already configured) before applying D8482
- cluster-argo
- archive-staging
- archive-production
vsellier updated the test plan for D8472: cassandra: Allow to configure the jmx for remote or local only access.
vsellier updated the diff for D8472: cassandra: Allow to configure the jmx for remote or local only access.
- rebase
- fix a typo on the jmxremote.access file name
- configure the jvm to use it
Sep 14 2022
Sep 14 2022
vsellier updated the test plan for D8479: staging-cluster: Declare the 2d disk on the local storage of uffizi.
vsellier requested review of D8479: staging-cluster: Declare the 2d disk on the local storage of uffizi.
It works \o/
rancher seems to create emptydir volume in /var/lib/kubelet, except the /var/lib/kubelet/pki directory, everything is ephemeral in this directory so we could easily use a partition backed by a local storage disk.
It will also remove an unecessary pressure on ceph for the pod relative data.
The /var/lib/docker directory could also be moved to this local partition as everything in docker can be lost.
I will manually try that on one staging node to check if it can work before changing the terraform / puppet code
vsellier committed R260:c76b657d58b2: graphql: Declare startup and liveness probes (authored by vsellier).
graphql: Declare startup and liveness probes
vsellier committed R260:aec926905060: graphql: change the sentry secret to optional (authored by vsellier).
graphql: change the sentry secret to optional
rebase
In order to test the local storage on nodes declared on uffizi, I configured a new scratch storage on this hypervisor.
Following T3707#73522 and https://pve.proxmox.com/wiki/Storage:_LVM_Thin
root@uffizi:~# lvcreate -L200G -n proxmox-scratch vg-louvre Logical volume "scratch" created.
vsellier changed the status of T4506: Use local hypervisor storage in the loader pods, a subtask of T4144: Elastic worker infrastructure, from Open to Work in Progress.
vsellier changed the status of T4506: Use local hypervisor storage in the loader pods from Open to Work in Progress.
vsellier added a subtask for T4523: Dynamic infrastructure: T4534: Evaluate MetalLB as inbound loadbalancer.
vsellier added a parent task for T4534: Evaluate MetalLB as inbound loadbalancer: T4523: Dynamic infrastructure.
vsellier requested review of D8472: cassandra: Allow to configure the jmx for remote or local only access.
vsellier closed T4510: [cassandra] Profile the replayer cpu consumption, a subtask of T4373: [cassandra] Test the new hardware, as Resolved.
I close this issue because after the @vlorentz 's analysis it seems there isn't a lot of things to improve
Sep 13 2022
Sep 13 2022
increase origin_visit_replayers
cassandra: redispatch replayers
These are the results of the different algorithms tests for the directory_add (with 20 directory replayers)
- one-by-one
vsellier raised the priority of T4455: Upgrade elk stack to a more recent version from Normal to High.
postgres=# select count(*) from pg_stat_activity where query like '%UNNEST(%';
count
-------
64
(1 row)postgres=# select count(*) from pg_stat_activity where query like '%UNNEST(%';
count
-------
64
(1 row)Sep 12 2022
Sep 12 2022
All the indexers were stopped at 20:00 FR because something was consummng all the bandwidth of the VPN between azure and the our infra.
root@pergamon:/etc/clustershell# clush -b -w @indexer-workers "puppet agent --disable 'stop indexer to avoid bandwith consumption'" root@pergamon:/etc/clustershell# clush -b -w @indexer-workers "systemctl stop swh-indexer-journal-client@*"
vsellier committed R260:7b43b64925da: test the batch directory add algorithm (authored by vsellier).
test the batch directory add algorithm
vsellier committed R260:275a980bb2bc: test the concurrent directory add algorithm (authored by vsellier).
test the concurrent directory add algorithm
vsellier committed rSPSITEaf3bd6139e3e: icinga: fix a typo on the graphql host declaration (authored by vsellier).
icinga: fix a typo on the graphql host declaration
vsellier committed rSPSITE527c69ff3e50: staging: Change the monitoring profile of db1 to sql (authored by vsellier).
staging: Change the monitoring profile of db1 to sql