This task is dedicated to start a discussion on the opportunity to replace part/all our (celery) task-based architecture for handling background processing of the archive.
First come in mind replacing the task-based indexers by (kafka) journal consumer ones. But we should consider the whole scheduler/celery/task model as well.
Also, we should not stick to kafka for now. Let's have in mind all the reasonable alternatives, with pros/cons of each one of them.
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T2217 Plumbings | ||
| Migrated | gitlab-migration | T2218 Orchestration | ||
| Migrated | gitlab-migration | T2063 Investigate stream-based execution model as a replacement for current scheduler/celery/task based approach | ||
| Migrated | gitlab-migration | T1521 Explore Faust as a possible kafka-based replacement for celery |
A few notes
NATS
NATS is an open-source messaging system written in Go. In the context of this discussion, we want to focus on NATS Streaming, which implements some/most of the basic requirements we need.
See for example:
https://storageos.com/nats-good-gotchas-awesome-features/
https://danielwertheim.se/nats-what-a-beautiful-protocol/
There is also a bunch of other tools using NATS as backbone:
Some related work : https://faust.readthedocs.io/en/latest/ (a Python stream processing environment inspired by KafkaStreams)