Page MenuHomeSoftware Heritage
Differential D8400

archive-staging: Deploy listers in cluster
ClosedPublic
Actions

Authored by ardumont on Sep 6 2022, 3:36 PM.

Details

Reviewers
vsellier
Group Reviewers
System administrators
Maniphest Tasks
T4144: Elastic worker infrastructure
Commits
R260:9ea373925c13: archive-staging: Deploy listers in cluster
Summary

It's starting small by adding an existing and small lister, the gnu one.

Related to T4144

Test Plan

Testing this with the gnu lister is ok [1]

helm install is not complaining [2]

[1]

17:46:41 swh-scheduler@db1:5432=> select * from task_run where task = (select id from task where type='list-gnu-full') order by ended desc limit 1;
+----------+----------+--------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------+----------+
|    id    |   task   |              backend_id              |           scheduled           |            started            |             ended             |                       metadata                        |  status  |
+----------+----------+--------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------+----------+
| 75294590 | 17271527 | 463a5b35-236a-4a23-87cb-63f0c9918b22 | 2022-09-06 15:42:28.566102+00 | 2022-09-06 15:42:50.948995+00 | 2022-09-06 15:42:57.389424+00 | {"worker": "celery@lister-gnu-full-7d798574f6-dsc7z"} | eventful |
+----------+----------+--------------------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------------------------------+----------+
(1 row)

Time: 6.620 ms

[2]

helm install --values values-swh-application-versions.yaml  --values swh/values/default.yaml --values swh/values/staging.yaml swh-test swh --dry-run --debug --set namespace=test 2>&1

Diff Detail

Repository
R260 Helm charts for swh packages
Branch
add-lister
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 31379
Build 49088: arc lint + arc unit

Event Timeline

ardumont created this revision.Sep 6 2022, 3:36 PM
Herald added a reviewer: System administrators. · View Herald TranscriptSep 6 2022, 3:36 PM
ardumont requested review of this revision.Sep 6 2022, 3:36 PM
Harbormaster completed remote builds in B31344: Diff 30314.Sep 6 2022, 3:36 PM
ardumont mentioned this in T4144: Elastic worker infrastructure.Sep 6 2022, 3:37 PM
This revision was not accepted when it landed; it landed in state Needs Review.Sep 6 2022, 4:48 PM
Closed by commit R260:9ea373925c13: archive-staging: Deploy listers in cluster (authored by ardumont). · Explain Why
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: R260:9ea373925c13: archive-staging: Deploy listers in cluster.
ardumont reopened this revision.Sep 6 2022, 4:49 PM

Nope, phab, it's in another branch!

ardumont edited the test plan for this revision. (Show Details)Sep 6 2022, 5:16 PM
vsellier requested changes to this revision.Sep 6 2022, 5:27 PM
vsellier added a subscriber: vsellier.
vsellier added inline comments.
swh/values/staging.yaml
114
This revision now requires changes to proceed.Sep 6 2022, 5:27 PM
ardumont updated this revision to Diff 30332.Sep 6 2022, 5:31 PM

Fix typo in scheduler

Harbormaster completed remote builds in B31359: Diff 30332.Sep 6 2022, 5:31 PM
ardumont edited the test plan for this revision. (Show Details)Sep 6 2022, 5:47 PM
ardumont updated this revision to Diff 30351.Sep 7 2022, 5:52 PM

Update

Harbormaster completed remote builds in B31379: Diff 30351.Sep 7 2022, 5:52 PM
vsellier accepted this revision.Sep 7 2022, 5:54 PM
This revision is now accepted and ready to land.Sep 7 2022, 5:54 PM
ardumont closed this revision.Sep 7 2022, 5:55 PM

Landed.

Revision Contents
Changeset List

PathSize
swh/
templates/
listers/
41 lines
119 lines
83 lines
values/
11 lines
33 lines
3 lines

Diff 30351

View Options

swh/templates/listers/configmap.yaml

  • This file was added.
{{ if .Values.listers.enabled -}}
{{- range $lister_type, $deployment_config := .Values.listers.deployments -}}
{{- $lister_name := ( print "lister-" $lister_type ) -}}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ $lister_name }}-template
namespace: {{ $.Values.namespace }}
data:
config.yml.template: |
storage:
cls: remote
url: http://{{ $.Values.listers.storage.host }}:{{ $.Values.listers.storage.port }}/
scheduler:
cls: remote
url: http://{{ $.Values.listers.scheduler.host }}:{{ $.Values.listers.scheduler.port }}/
celery:
task_broker: ${host}
task_queues:
{{- range $queue := get $deployment_config "queues" }}
- {{ $queue }}
{{- end }}
credentials:
init-container-entrypoint.sh: |
#!/bin/bash
set -e
CONFIG_FILE=/etc/swh/config.yml
# substitute environment variables when creating the default config.yml
eval echo \""$(</etc/swh/configuration-template/config.yml.template)"\" \
> $CONFIG_FILE
CREDS_LISTER_PATH=/etc/credentials/listers/credentials
[ -f $CREDS_LISTER_PATH ] && \
sed 's/^/ /g' $CREDS_LISTER_PATH >> $CONFIG_FILE
{{ end }}
{{- end -}}
View Options

swh/templates/listers/deployment.yaml

  • This file was added.
{{ if .Values.listers.enabled -}}
{{- $configurationChecksum := include (print $.Template.BasePath "/listers/configmap.yaml") . -}}
{{- range $lister_type, $deployment_config := .Values.listers.deployments -}}
{{- $lister_name := ( print "lister-" $lister_type ) -}}
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: {{ $lister_name }}
namespace: {{ $.Values.namespace }}
labels:
app: {{ $lister_name }}
spec:
selector:
matchLabels:
app: {{ $lister_name }}
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
template:
metadata:
labels:
app: {{ $lister_name }}
annotations:
# Force a rollout upgrade if the configuration changes
checksum/config: {{ $configurationChecksum | sha256sum }}
spec:
{{- if $.Values.listers.affinity }}
affinity:
{{ toYaml $.Values.listers.affinity | nindent 8 }}
{{- end }}
initContainers:
- name: prepare-configuration
image: debian:bullseye
imagePullPolicy: Always
envFrom:
- secretRef:
# define the ${host) values for configuration "celery > task_broker: " key
name: amqp-access-credentials
command:
- /entrypoint.sh
volumeMounts:
- name: configuration-template
mountPath: /entrypoint.sh
subPath: "init-container-entrypoint.sh"
readOnly: true
- name: configuration
mountPath: /etc/swh
- name: configuration-template
mountPath: /etc/swh/configuration-template
- name: lister-credentials-secrets
mountPath: /etc/credentials/listers
readOnly: true
containers:
- name: listers
resources:
requests:
memory: {{ get $deployment_config "requestedMemory" | default "512Mi" }}
cpu: {{ get $deployment_config "requestedCpu" | default "500m" }}
limits:
memory: "4000Mi"
cpu: "1200m"
image: {{ $.Values.swh_listers_image }}:{{ $.Values.swh_listers_image_version }}
imagePullPolicy: Always
command:
- /bin/bash
args:
- -c
- /opt/swh/entrypoint.sh
lifecycle:
preStop:
exec:
command: ["kill", "1"]
env:
- name: STATSD_HOST
value: {{ $.Values.statsdExternalHost | default "prometheus-statsd-exporter" }}
- name: STATSD_PORT
value: {{ $.Values.statsdPort | default "9125" | quote }}
- name: CONCURRENCY
value: {{ get $deployment_config "concurrency" | default 1 | quote }}
- name: MAX_TASKS_PER_CHILD
value: {{ get $deployment_config "maxTasksPerChild" | default 1 | quote }}
- name: LOGLEVEL
value: {{ get $deployment_config "logLevel" | default "INFO" | quote }}
- name: SWH_CONFIG_FILENAME
value: /etc/swh/config.yml
- name: SWH_SENTRY_ENVIRONMENT
value: {{ $.Values.sentry.environment }}
- name: SWH_MAIN_PACKAGE
value: {{ $.Values.listers.sentrySwhPackage }}
- name: SWH_SENTRY_DSN
valueFrom:
secretKeyRef:
name: lister-sentry-secrets
key: sentry-dsn
# 'name' secret must exist & include key "host"
optional: false
volumeMounts:
- name: configuration
mountPath: /etc/swh
volumes:
- name: configuration
emptyDir: {}
- name: configuration-template
configMap:
name: {{ $lister_name }}-template
defaultMode: 0777
items:
- key: "config.yml.template"
path: "config.yml.template"
- key: "init-container-entrypoint.sh"
path: "init-container-entrypoint.sh"
- name: lister-credentials-secrets
secret:
secretName: lister-credentials-secrets
optional: true
{{ end }}
{{- end -}}
View Options

swh/templates/listers/keda-autoscaling.yaml

  • This file was added.
{{ if .Values.listers.enabled -}}
{{- range $lister_type, $deployment_config := .Values.listers.deployments -}}
{{ if get $deployment_config "autoScaling" }}
{{- $autoscalingConfig := get $deployment_config "autoScaling" -}}
{{- $lister_name := ( print "lister-" $lister_type ) -}}
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: amqp-authentication
namespace: {{ $.Values.namespace }}
spec:
secretTargetRef: # Optional.
- parameter: host
name: amqp-access-credentials
key: host
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: {{ $lister_name }}-operators
namespace: {{ $.Values.namespace }}
spec:
scaleTargetRef:
apiVersion: apps/v1 # Optional. Default: apps/v1
kind: Deployment # Optional. Default: Deployment
# Mandatory. Must be in same namespace as ScaledObject
name: {{ $lister_name }}
# envSourceContainerName: {container-name} # Optional. Default:
# .spec.template.spec.containers[0]
pollingInterval: 30 # Optional. Default: 30 seconds
cooldownPeriod: 300 # Optional. Default: 300 seconds
idleReplicaCount: 0 # Optional. Must be less than
# minReplicaCount
minReplicaCount: {{ get $autoscalingConfig "minReplicaCount" | default 0 }}
maxReplicaCount: {{ get $autoscalingConfig "maxReplicaCount" | default 5 }}
fallback: # Optional. Section to specify fallback
# options
failureThreshold: 3 # Mandatory if fallback section is
# included
replicas: 6 # Mandatory if fallback section is
# included
advanced: # Optional. Section to specify advanced
# options
restoreToOriginalReplicaCount: false # Optional. Default: false
horizontalPodAutoscalerConfig: # Optional. Section to specify HPA
# related options
behavior: # Optional. Use to modify HPA's scaling
# behavior
scaleDown:
stabilizationWindowSeconds: 60 # default 300
policies:
- type: Percent
value: 2
periodSeconds: 15
triggers:
{{- range $queue := get $deployment_config "queues" }}
- type: rabbitmq
authenticationRef:
name: amqp-authentication
metadata:
host: host
# Optional. If not specified, it must be done
# by using TriggerAuthentication.
protocol: auto # Optional. Specifies protocol to use,
# either amqp or http, or auto to
# autodetect based on the `host` value.
# Default value is auto.
mode: QueueLength # QueueLength or MessageRate
# message backlog or publish/sec.
# target per instance
value: {{ get $autoscalingConfig "queueThreshold" | default 100 | quote }}
queueName: {{ $queue }}
vhostName: / # Optional. If not specified, use the vhost in the
# `host` connection string. Alternatively, you can
# use existing environment variables to read
# configuration from: See details in "Parameter
# list" section hostFromEnv: RABBITMQ_HOST%
{{- end }}
{{ end }}
{{ end }}
{{- end -}}
View Options

swh/values/default.yaml

Show All 24 Lines nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution: requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms: nodeSelectorTerms:
- matchExpressions: - matchExpressions:
- key: "swh/loader" - key: "swh/loader"
operator: In operator: In
values: values:
- "true" - "true"
listers:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "swh/lister"
operator: In
values:
- "true"
graphql: graphql:
affinity: affinity:
nodeAffinity: nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution: requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms: nodeSelectorTerms:
- matchExpressions: - matchExpressions:
- key: swh/rpc - key: swh/rpc
operator: In operator: In
values: values:
- "true" - "true"
statsd_exporter: statsd_exporter:
enabled: true enabled: true
View Options

swh/values/staging.yaml

Loading...
View Options

values-swh-application-versions.yaml

Loading...
About · Developer information · Legal