RabbitMQ is a resource hog and is currently on the moma virtual machine.
Moving it to a dedicated machine will increase reliability of the webapp, and free some resources to allow more workers on the webapp.
Description
Description
| Status | Assigned | Task | ||
|---|---|---|---|---|
| Migrated | gitlab-migration | T614 Open up the Software Heritage API to the world | ||
| Migrated | gitlab-migration | T653 Split the scheduler (rabbitmq) out of the moma virtual machine |
Event Timeline
Comment Actions
The new server saatchi.internal.softwareheritage.org has been created. RabbitMQ data is being synced.
Comment Actions
It turns out that rabbitmq doesn't allow you to just move the data directory: it's a dump of the erlang data structure and that depends on the hostname of the current node.
I therefore started the rabbitmq server on saatchi and clustered it with the one on moma. I set it up so that the queues get mirrored.
However, the synchronization is SLOW as ice. 50 messages per second slow.
The current status is that all queues are synced except for swh_loader_git, which contains 37+ million messages.
I'm tempted to just drop the messages and start fresh, and the scheduler will handle the rest.
Thoughts?
Comment Actions
Done:
- stopped rabbitmq on moma
- disabled rabbitmq on moma
- removed rabbit@moma from the cluster on saatchi
- updated the CNAME rabbitmq to point to saatchi
- marked the relevant tasks as "lost" in the scheduler