Page MenuHomeSoftware Heritage

Split the scheduler (rabbitmq) out of the moma virtual machine
Closed, MigratedEdits Locked

Description

RabbitMQ is a resource hog and is currently on the moma virtual machine.

Moving it to a dedicated machine will increase reliability of the webapp, and free some resources to allow more workers on the webapp.

Event Timeline

olasd moved this task from Restricted Project Column to Restricted Project Column on the Restricted Project board.

The new server saatchi.internal.softwareheritage.org has been created. RabbitMQ data is being synced.

It turns out that rabbitmq doesn't allow you to just move the data directory: it's a dump of the erlang data structure and that depends on the hostname of the current node.

I therefore started the rabbitmq server on saatchi and clustered it with the one on moma. I set it up so that the queues get mirrored.

However, the synchronization is SLOW as ice. 50 messages per second slow.

The current status is that all queues are synced except for swh_loader_git, which contains 37+ million messages.

I'm tempted to just drop the messages and start fresh, and the scheduler will handle the rest.

Thoughts?

Let's start fresh.

Oh, rabbitmq, how much we love you!

Done:

  • stopped rabbitmq on moma
  • disabled rabbitmq on moma
  • removed rabbit@moma from the cluster on saatchi
  • updated the CNAME rabbitmq to point to saatchi
  • marked the relevant tasks as "lost" in the scheduler