Feed Advanced Search
Sep 12 2022
Sep 12 2022
vsellier committed R260:649738df8042: Measure performance for one-by-one directory replayer (authored by vsellier).
Measure performance for one-by-one directory replayer
Support specific options per replayer
vsellier committed R260:8c8dc14580fc: Merge remote-tracking branch 'origin/master' into cassandra (authored by vsellier).
Merge remote-tracking branch 'origin/master' into cassandra
vsellier requested review of D8449: staging: Change the monitoring profile of db1 to sql.
Sep 9 2022
Sep 9 2022
vsellier added a comment to T4458: Test reaper to automate the cassandra repair actions.
reaper access the cassandra server through jmx. The cassandra deployment scripts need to be adapted (in progress) to expose jmx on the public interface.
When publicly exposed, the cassandra startup scripts force to password protect the jmx accesses.
vsellier added a comment to T4458: Test reaper to automate the cassandra repair actions.
A new production node for replayers and generic load was added on the cluster to add more compute resources to allow testing the tool
vsellier changed the status of T4458: Test reaper to automate the cassandra repair actions, a subtask of T4373: [cassandra] Test the new hardware, from Open to Work in Progress.
vsellier changed the status of T4458: Test reaper to automate the cassandra repair actions from Open to Work in Progress.
vsellier committed R260:7daee55ef56d: add an affinity of the replayers to the nodes with swh/replayer=true (authored by vsellier).
add an affinity of the replayers to the nodes with swh/replayer=true
vsellier committed rSPRE943c46f87c61: rancher-production: Add a new node on hypervisor3 with 6 cpus (authored by vsellier).
rancher-production: Add a new node on hypervisor3 with 6 cpus
vsellier added a comment to T4510: [cassandra] Profile the replayer cpu consumption.
here some profiling of a couple of replayers:
vsellier committed R260:a45bdce2eb24: Merge remote-tracking branch 'origin/master' into cassandra (authored by vsellier).
Merge remote-tracking branch 'origin/master' into cassandra
vsellier committed R260:f64a9a9829cd: Try to reduce the global cpu consumption to reduce the hypervisor load (authored by vsellier).
Try to reduce the global cpu consumption to reduce the hypervisor load
Sep 8 2022
Sep 8 2022
vsellier changed the status of T4510: [cassandra] Profile the replayer cpu consumption from Open to Work in Progress.
vsellier changed the status of T4510: [cassandra] Profile the replayer cpu consumption, a subtask of T4373: [cassandra] Test the new hardware, from Open to Work in Progress.
vsellier closed T4509: [swh-graph] Configure the max_memory to use, a subtask of T4507: Out of memory on granet, as Resolved.
diff landed and deployed, graph restarted
vsellier committed rSPSITE303c48250b95: swh-graph: configure the max heap allocated to the java backend (authored by vsellier).
swh-graph: configure the max heap allocated to the java backend
vsellier updated the diff for D8431: swh-graph: configure the max heap allocated to the java backend.
rebase
vsellier added a comment to T4509: [swh-graph] Configure the max_memory to use.
I forgot to mention, it seems expect during some peak, the used memory is around 350Go
vsellier requested review of D8431: swh-graph: configure the max heap allocated to the java backend.
vsellier committed R260:7f552eb93182: reduce concurrent loaders for origin, increase directory (authored by vsellier).
reduce concurrent loaders for origin, increase directory
vsellier added a comment to T4330: Deploy maven stack in production.
\o/ great
vsellier changed the status of T4509: [swh-graph] Configure the max_memory to use, a subtask of T4507: Out of memory on granet, from Open to Work in Progress.
vsellier changed the status of T4509: [swh-graph] Configure the max_memory to use from Open to Work in Progress.
vsellier committed R260:097e68bf3067: Merge remote-tracking branch 'origin/master' into cassandra (authored by vsellier).
Merge remote-tracking branch 'origin/master' into cassandra
Adapt replayer dispatching
vsellier committed rSPREecec22a1a3a2: thanos: Increase the allocated memory to avoid OOM killer (authored by vsellier).
thanos: Increase the allocated memory to avoid OOM killer
vsellier requested review of D8415: thanos: Increase the allocated memory to avoid OOM killer.
vsellier added a comment to T4507: Out of memory on granet.
@vlorentz I assigned the task to you because if I'm not wrong you are running some experiments on granet.
I don't know what, but you should be more gentle with the server
vsellier committed R260:a15da28adfaa: reduce origin comumption as most of the partition are replayed (authored by vsellier).
reduce origin comumption as most of the partition are replayed
speed up origin topic replay
Sep 7 2022
Sep 7 2022
vsellier updated the task description for T4506: Use local hypervisor storage in the loader pods.
vsellier updated the task description for T4506: Use local hypervisor storage in the loader pods.
vsellier committed R260:08c8caf5c557: Merge remote-tracking branch 'origin/master' into cassandra (authored by vsellier).
Merge remote-tracking branch 'origin/master' into cassandra
stabilize the number of replayer
fix comment start
vsellier committed R260:4b88963f3360: remove empty deployment as 0 is considered as empty values (authored by vsellier).
remove empty deployment as 0 is considered as empty values
try to fix 0 replicas deployments
vsellier committed R260:89c0b457ad2f: prioritize small topics to free resources for bigger ones later (authored by vsellier).
prioritize small topics to free resources for bigger ones later
Sep 6 2022
Sep 6 2022
vsellier updated the task description for T4479: uncouple the java grpc server from the python HTTP server.
yes even better
vsellier committed rSPREc4eec83f84e9: money: fix chromium issue with missing sse3 instructions (authored by vsellier).
money: fix chromium issue with missing sse3 instructions
vsellier requested changes to D8400: archive-staging: Deploy listers in cluster.
vsellier requested review of D8405: money: fix chromium issue with missing sse3 instructions.
vsellier added inline comments to D8397: Deploy maven-exporter production node.
vsellier added a comment to T4497: [sentry] Out of disk space.
The root cause is a swh-graph experiment that generated a lot of grpc errors which are huge.
vsellier added a comment to T4497: [sentry] Out of disk space.
No consumers seem to have a big lag on these topics, so it should be possible to reduce the lag to unblock the server and have a look which service is sending the events:
root@riverside:/var/lib/sentry-onpremise# docker-compose-1.29.2 run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --list | tr -d '\r' | xargs -t -n1 docker-compose-1.29.2 run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --describe --group | grep -e GROUP -e " events " Creating sentry-self-hosted_kafka_run ... done docker-compose-1.29.2 run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --describe --group snuba-consumers Creating sentry-self-hosted_kafka_run ... done GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID snuba-consumers events 0 82585390 82587094 1704 - - - docker-compose-1.29.2 run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --describe --group snuba-post-processor:sync:6fa9928e1d6911edac290242ac170014 Creating sentry-self-hosted_kafka_run ... done GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID docker-compose-1.29.2 run --rm kafka kafka-consumer-groups --bootstrap-server kafka:9092 --describe --group ingest-consumer Creating sentry-self-hosted_kafka_run ... done
vsellier added a comment to T4497: [sentry] Out of disk space.
The biggest topics are:
root@riverside:/var/lib/docker/volumes/sentry-kafka/_data# du -sch * | sort -h | tail -n 5 31M snuba-commit-log-0 291M outcomes-0 30G ingest-events-0 43G events-0 73G total
vsellier committed R260:9b4840fdba24: Reduce cpu and memory reservation for a couple of topics when the lag recovered (authored by vsellier).
Reduce cpu and memory reservation for a couple of topics when the lag recovered
Sep 5 2022
Sep 5 2022
adapt cpu request for snapshots
vsellier committed R260:69f2df7d9a89: Adjust memory and cpu request according real behavior (authored by vsellier).
Adjust memory and cpu request according real behavior
Merge branch 'master' into cassandra
vsellier updated the task description for T4474: onboarding Lunar.
wip - Configure the replayers
vsellier updated the task description for T4474: onboarding Lunar.
vsellier updated the task description for T4474: onboarding Lunar.
Declare lunar credentials
add password for lunar
vsellier requested review of D8391: Declare lunar credentials.
vsellier committed rSPRE445014116634: Declare the archive-production k8s cluster (authored by vsellier).
Declare the archive-production k8s cluster
Sep 4 2022
Sep 4 2022
vsellier committed rSKCONF1ad68a77cc9a: Deploy the cassandra replayers on the archive production cluster (authored by vsellier).
Deploy the cassandra replayers on the archive production cluster
Sep 2 2022
Sep 2 2022
vsellier committed rSKCONF26d8c509c2e8: Declare archive-production cluster applications (authored by vsellier).
Declare archive-production cluster applications
vsellier committed rSKCONF3ebfe35fde48: Delete the production-cassandra cluster configuration (authored by vsellier).
Delete the production-cassandra cluster configuration
vsellier committed rSPSITE8679da780d99: Declare the missing read-only objstorage dns entry (authored by vsellier).
Declare the missing read-only objstorage dns entry
vsellier requested review of D8380: Declare the missing read-only objstorage dns entry.
Sep 1 2022
Sep 1 2022
fix hiera file names for hosts
vsellier committed rSPSITE81197e23e261: Remove local dns proxy of production rancher nodes (authored by vsellier).
Remove local dns proxy of production rancher nodes
Aug 31 2022
Aug 31 2022
vsellier committed rSPSITE33cd4db917f6: staging: Reduce the worker restart overhead (authored by vsellier).
staging: Reduce the worker restart overhead
vsellier committed rSPSITEceb1d7760989: staging: Increase the number of workers for storage and indexer storage (authored by vsellier).
staging: Increase the number of workers for storage and indexer storage
vsellier updated the diff for D8371: staging: Increase the number of workers for storage and indexer storage.
rebase
vsellier updated the test plan for D8371: staging: Increase the number of workers for storage and indexer storage.
vsellier updated the diff for D8371: staging: Increase the number of workers for storage and indexer storage.
Increase the number of request handled by a storage worker before it's restarted
vsellier requested review of D8371: staging: Increase the number of workers for storage and indexer storage.
vsellier committed rSPRE1614bb4f4b5b: terraform: centralize the production bridge configuration (authored by vsellier).
terraform: centralize the production bridge configuration