zfs dataset created on storage1:

root@storage1:~# zfs create -o mountpoint=/srv/kafka -o atime=off -o relatime=on data/kafka
root@storage1:~# zfs list
NAME           USED  AVAIL     REFER  MOUNTPOINT
data          5.39T  21.0T       96K  /data
data/kafka      96K  21.0T       96K  /srv/kafka
data/objects  5.38T  21.0T     5.38T  /srv/softwareheritage/objects

When D6477 will be validated and applied, the move action will be:

rebase

update according to the feedbacks

Some flame graphs of storage was performed during the ingestion with 50 workers in //

Some runs with the fix:
It globally improves the stability of the benchmark by reducing the timeouts.

Remove an unnecessary linefeed

solved by T3487#71814

solved by T3487#71814

Sure, we should have authentication / rate limit on this.
But if I'm not wrong, the target is to test the mirroring with ENEA.
If we add authentication, we need to improve the objstorage-replayer / objstorage to support it.

rSENV646f62805ef564bceed4d3a4d84d8fb6890f2d19 declares the new certificate for the vagrant tests (wrong task on the commit message)

update commit message

• factorize the exported configuration
• use the right exporter port on met

rebase

I think the issue can be closed.
The pros are:

• it simplify the cluster management (create, configuration and most of all, kubernetes upgrades)
• centralize the global view of the cluster and what is running on it
• OSS and transparent policy

The propose plan looks too naive as some zk configuration also needs to be updated.
A recommended way is to add a new node on the cluster, migrate the partitions on the new node and shutdown the old one.
Doing like this ensure all the data and configuration will be correctly migrated, and without downtime, which is not negligible as the mirror tests of enea are in progress.

The loader were finally stabilized. It was due to a wrong celery configuration.
Changing the pool configuration from solo to prefork solved the problem even if the concurrency is kept to one.
Solo looked indicated in environment like the POC but for obvious reasons, it was not working as expected:

credentials added on the credential database under the refs:

• operations/kafka/credentials/staging/swh-enea
• operations/kafka/credentials/production/swh-enea

Production credentials created:

+ export zookeeper_servers=kafka1.internal.softwareheritage.org:2181
+ zookeeper_servers=kafka1.internal.softwareheritage.org:2181
+ export bootstrap_servers=kafka1.internal.softwareheritage.org:9092
+ bootstrap_servers=kafka1.internal.softwareheritage.org:9092
+ '[' -z swh-enea -o -z redacted ']'
+ set -eu
+ /opt/kafka/bin/kafka-configs.sh --zookeeper kafka1.internal.softwareheritage.org:2181/kafka/softwareheritage --alter --add-config 'SCRAM-SHA-256=[iterations=8192,password=redacted],SCRAM-SHA-512=[password=redacted]' --entity-type users --entity-name swh-enea
Warning: --zookeeper is deprecated and will be removed in a future version of Kafka.
Use --bootstrap-server instead to specify a broker to connect to.
Completed updating config for entity: user-principal 'swh-enea'.
+ /opt/kafka/bin/kafka-acls.sh --bootstrap-server kafka1.internal.softwareheritage.org:9092 --add --resource-pattern-type PREFIXED --topic swh.journal.objects. --allow-principal User:swh-enea --operation READ
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)

Credentials create in stagingd:

ACLs for principal `User:swh-enea`
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=swh.journal.objects., patternType=PREFIXED)`: 
 	(principal=User:swh-enea, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:swh-enea, host=*, operation=READ, permissionType=ALLOW)

Actions to perform for the migration:

• add the role::swh_kafka_broker role to the new server in puppet and deploy
• stop the staging workers
• stop kafka on journal0
• rsync the content from journal0 to the new server
• update the staging configurations to use the new server as journal
• update the NAT public staging for staging journal on the firewall (https://192.168.50.1/firewall_nat.php)

LGTM thanks

keda looks promising. P1193 is an example of configuration working for the docker environment. It's able to scale to 0 when no messages are present on the queue.
When messages are present, the loaders are launched progressively until the limit of cpu/memory of the host is reached or the max number of allowed worker is reached.

intermediary status:

• the bench lab is easily deployable on g5k on several workers to distribute the load [1]
• it's working well when the load is not so high. When the number of worker is increased, it seems the workers have some issues to talk with rabbitmq:

[loaders-77cdd444df-flcv9 loaders] [2021-09-30 23:46:55,449: INFO/MainProcess] missed heartbeat from celery@loaders-77cdd444df-p9ds5                                                                                                                                                     
[loaders-77cdd444df-flcv9 loaders] [2021-09-30 23:46:55,449: INFO/MainProcess] missed heartbeat from celery@loaders-77cdd444df-n6pvm                    
[loaders-77cdd444df-flcv9 loaders] [2021-09-30 23:46:55,449: INFO/MainProcess] missed heartbeat from celery@loaders-77cdd444df-mrcjj                                                                                                                                                     
[loaders-77cdd444df-flcv9 loaders] [2021-09-30 23:46:55,449: INFO/MainProcess] missed heartbeat from celery@loaders-77cdd444df-7bn4s                                                                                       
[loaders-77cdd444df-flcv9 loaders] [2021-09-30 23:46:55,449: INFO/MainProcess] missed heartbeat from celery@loaders-77cdd444df-lg2bd

and also an unexplained time drift:

[loaders-77cdd444df-flcv9 loaders] [2021-09-30 23:46:55,447: WARNING/MainProcess] Substantial drift from celery@loaders-77cdd444df-lxjpl may mean clocks are out of sync.  Current drift is 
[loaders-77cdd444df-flcv9 loaders] 356 seconds.  [orig: 2021-09-30 23:46:55.447181 recv: 2021-09-30 23:40:59.633444]                                                                                                                                                                     
[loaders-77cdd444df-flcv9 loaders]                                                                                                                                                                            
[loaders-77cdd444df-flcv9 loaders] [2021-09-30 23:46:55,447: WARNING/MainProcess] Substantial drift from celery@loaders-77cdd444df-jd6v9 may mean clocks are out of sync.  Current drift is                                                                                              
[loaders-77cdd444df-flcv9 loaders] 355 seconds.  [orig: 2021-09-30 23:46:55.447552 recv: 2021-09-30 23:41:00.723983]                                  
[loaders-77cdd444df-flcv9 loaders]

Intermediary status:

• We have successfully ran loaders in staging using the helm chart we have wrote [1] and an hardcoded number of worker, It adds the possibility to perform rolling upgrades for example
• We have tried the integrated horizontal pod autoscaler [2], it works pretty well but it's not adapted for our worker scenario. It's based on the cpu consumption(on our test [3], but can be other things) of the pod to decide if the number of running pods must be upscaled or downscaled. It can be very useful to manage classical load like for gunicorn container, but not for the scenario of long running tasks
• Kubernetes also has some functionalities to reduce the pressure on a node when some limts are reached but it looks like it's more emergency actions than proper scaling management. It's configured at the kubelet level and not dynamic at all [4]. It was rapidly tested but we have lost the node due to oom before the node eviction starts.

cc @seirl

After having hard time, we have solved several issues:

• The rancher initialization problem was because we were using a wrong version of k3s compared to the compatibility matrix of rancher.

We installed rancher 2.5.9 on a recent version of k3s installing kubernetes 1.22.2. According to the compatibility matrix of rancher[1], using a older version of k3s solved the problem and the clusters start correctly after that

rebase