Page MenuHomeSoftware Heritage

staging: Autoscaling of the indexers not properly configured
Closed, MigratedEdits Locked

Description

Seen in the archive-staging cluster's events:

swh         0s          Warning   KEDAScalerFailed             scaledobject/indexer-origin-extrinsic-scaledobject            error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh         0s          Warning   ScaledObjectCheckFailed      scaledobject/indexer-origin-extrinsic-scaledobject            Failed to ensure HPA is correctly created for ScaledObject
swh         0s          Warning   KEDAScalerFailed             scaledobject/indexer-content-mimetype-scaledobject            error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh         0s          Warning   ScaledObjectCheckFailed      scaledobject/indexer-content-mimetype-scaledobject            Failed to ensure HPA is correctly created for ScaledObject
swh         0s          Warning   KEDAScalerFailed             scaledobject/indexer-content-fossology-license-scaledobject   error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
swh         0s          Warning   ScaledObjectCheckFailed      scaledobject/indexer-content-fossology-license-scaledobject   Failed to ensure HPA is correctly created for ScaledObject
swh         0s          Warning   KEDAScalerFailed             scaledobject/indexer-origin-intrinsic-scaledobject            error creating kafka client: kafka: client has run out of available brokers to talk to: dial tcp: address journal1.internal.staging.swh.network: missing port in address
❯ kubectl get scaledobject --context archive-staging -n swh | grep -e "^NAME" -e indexer                                                                                                    10:09:58
NAME                                             SCALETARGETKIND      SCALETARGETNAME                     MIN   MAX   TRIGGERS   AUTHENTICATION                                 READY   ACTIVE    FALLBACK   AGE
indexer-content-fossology-license-scaledobject   apps/v1.Deployment   indexer-content-fossology-license   1     2     kafka      keda-storage-replayer-trigger-authentication   False   Unknown   Unknown    14d
indexer-content-mimetype-scaledobject            apps/v1.Deployment   indexer-content-mimetype            1     2     kafka      keda-storage-replayer-trigger-authentication   False   Unknown   Unknown    14d
indexer-origin-extrinsic-scaledobject            apps/v1.Deployment   indexer-origin-extrinsic            1     2     kafka      keda-storage-replayer-trigger-authentication   False   Unknown   Unknown    14d
indexer-origin-intrinsic-scaledobject            apps/v1.Deployment   indexer-origin-intrinsic            1     2     kafka      keda-storage-replayer-trigger-authentication   False   Unknown   Unknown    14d

^ Ready: false Active:Unkown

Apparently, keda allows to monitor it's status through PodMonitor objects, we need to check how to activate it and to alert on this

Related to T4540

Event Timeline

vsellier triaged this task as Normal priority.Oct 12 2022, 10:11 AM
vsellier created this task.
vsellier renamed this task from staging: Autoscaling of the indexer not properly well configurred to staging: Autoscaling of the indexers not properly configured.Oct 12 2022, 10:14 AM
vsellier updated the task description. (Show Details)