Making RabbitMQ Recover from (a)Mnesia

In the company I work for we’re using RabbitMQ to offload non-timecritical processing of tasks. To be able to recover in case RabbitMQ goes down our queues are durable and all our messages are marked as persistent. We generally have a very low number of messages in flight at any moment in time. There’s just one queue with a decent amount of them: the “failed messages” dump.

The Problem

It so happens that after a botched update to the most recent version of RabbitMQ (3.5.3 at the time) our admins had to nuke the server and install it from scratch. They had made a backup of RabbitMQ’s Mnesia database and I was tasked to recover the messages from it.
This is the story of how I did it.

Since our RabbitMQ was configured to persist all the messages this should be generally possible. Surely I wouldn’t be the first one to attempt this. 😐

Looking through the Internet it seems there’s no way of ex/importing a node’s configuration if it’s not running. I couldn’t find any documentation on how to import a Mnesia backup into a new node or extract data from it into a usable form. 😞

The Idea

My idea was to setup a virtual machine (running Debian Wheezy) with RabbitMQ and then to somehow make it read/recover and run the broken server’s database.

In the following you’ll see the following placeholders:

  • RABBITMQ_MNESIA_BASE will be /var/lib/rabbitmq/mnesia  on Debian (see RabbitMQ’S file locations)
  • RABBITMQ_MNESIA_DIR is just $RABBITMQ_MNESIA_BASE/$RABBITMQ_NODENAME
  • BROKEN_NODENAME the $RABBITMQ_NODENAME of the broken server we have backups from
  • BROKEN_HOST the hostname of said server

One more thing before we start: if I say “fix permissions” below I mean

1st Try

My first try was to just copy the broken node’s Mnesia files to the VM’s $RABBITMQ_MNESIA_DIR failed. The files contained node names that RabbitMQ tried to reach but were unreachable from the VM.

So I tried to be a little bit more picky on what I copied.

First I had to reset $RABBITMQ_MNESIA_DIR by deleting it and have RabbitMQ recreate it. (I needed to do this way too many times 😭)

Stopping RabbitMQ I tried to feed it the broken server’s data in piecemeal fashion. This time I only copied the rabbit_*.[DCD,DCL]  and restarted RabbitMQ.

RabbitMQ Management Interface lists all the queues, but the node it thinks they're on is "down"
RabbitMQ’s management interface lists all the queues, but it thinks the node they’re on is “down”

Looking at the web management interface there were all the queues we were missing, but they were “down” and clicking on them told you

The object you clicked on was not found; it may have been deleted on the server.

Copying any more data didn’t solve the issue. So this was a dead end. 😫

2nd Try

So I thought why doesn’t the RabbitMQ in the VM pretend to be the exact same node as on the broken server?

So I created a /etc/rabbitmq/rabbitmq-env.conf  with NODENAME=$BROKEN_NODENAME  in there.

I copied the backup to $RABBITMQ_MNESIA_DIR (now with the new node name) and fixed the permissions.

Now starting RabbitMQ failed with

I edited /etc/hosts  to add $BROKEN_HOST to the list of names that resolve to 127.0.0.1.

Now restarting RabbitMQ failed with yet another error:

Now what? Why don’t I try to give it the Mnesia files piece by piece again?

  • Reset $RABBITMQ_MNESIA_DIR
  • Stop RabbitMQ
  • Copy rabbit_*  files in again and fix their permissions
  • Start RabbitMQ

All our queues were back and all their configuration seemed OK as well. But we still didn’t have our messages back yet.

RabbitMQ Data Recovery Screen Shot 2 - Node Up, Queues Empty
The queues have been restored, but they have no messages in them

Solution

So I tried to copy more and more files over from the backup repeating the above steps. I finally reached my goal after copying rabbit_* , msg_store_* , queues  and recovery.dets. Fixing their permissions and starting RabbitMQ it had all the queues restored with all the messages in them. 😂

RabbitMQ Data Recovery Screen Shot 3 - Messages Restored
Queues and messages restored

Now I could use ordinary methods to extract all the messages. Dumping all the messages and examining them they looked OK. Publishing the recovered messages to the new server I was pretty euphoric. 😁

2 thoughts on “Making RabbitMQ Recover from (a)Mnesia”

  1. This was a while back… but could you say a little more clearly what worked. Is it that you copied the listed files (rabbit_*, msg_store_*, queues) over the directory that you allowed RabbitMQ to create. There’s no recovery.dets file in my RabbitMQ. 

    1. The solution in a nutshell was:

      # pretend the new node is the old node
      vim /etc/rabbitmq/rabbitmq-env.conf
      # stop RabbitMQ
      sudo service rabbitmq-server stop
      # remove its database
      rm -r $RABBITMQ_MNESIA_DIR
      # let RabbitMQ create a fresh database for the node
      sudo service rabbitmq-server start
      # shut down RabbitMQ so we can manipulate the files safely
      sudo service rabbitmq-server stop
      # copy over backed-up files
      cp rabbit_* msg_store_* queues recovery.dets $RABBITMQ_MNESIA_DIR
      # start RabbitMQ and hope it can restore the data
      sudo service rabbitmq-server start

      If RabbitMQ came up correctly you can now use the standard methods (web interface, API, etc.) to do a proper backup of the database/configuration.

      I must emphasize that our configuration used durable queues and solely persistent messages (i.e. delivery_mode = 2 ).

      After I was able to restore the data I tried to find the minimal set of files that would allow RabbitMQ to recover the data. IIRC that’s how I came up with the list of files to copy from the backup to the new node.

      A short Internet search suggests recovery.dets is created when the RabbitMQ server hasn’t shut down cleanly. But I couldn’t tell you if it was necessary for the recovery of the Mnesia database to work.

Leave a Reply

Your email address will not be published. Required fields are marked *