Moving LXD Containers From One Pool to Another

When I started playing with LXD I just accepted the default storage configuration which creates an image file and uses that to initialize a ZFS pool. Since I’m using ZFS as my main file system this seemed silly as LXD can use an existing dataset as a source for a storage pool. So I wanted to migrate my existing containers to the new storage pool.

Although others seemed to to have the same problem there was no ready answer. Digging through the documentation I finally found out that the lxc move  command had a  -s  option … I had an idea. 💡 Here’s what I came up with …

Preparation

First we create the dataset on the existing ZFS pool and add it to LXC.

lxc storage list should show something like this now:

pool1 is the old pool backed by the image file and is used by some containers at the moment as can be seen in the “Used By” column.  pool2 is added by not used by any contaiers yet.

Moving

We now try to move our containers to pool2.

We can check with  lxc storage list whether we succeeded.

Indeed  pool2 is beeing used now. 😀 Just to be sure we check that zfs list -r mypool/lxd  also reflects this.

Awesome!

⚠ Note that this only moves the container, but not the LXC image it was cloned off of.

We can repeat this until all containers we care about are moved over to pool2.

Cleanup

To prevent new containers to use pool1  we have to edit the default  profile.

Finally …. when we’re happy with the migration and we’ve verified that everything works as expected we can now remove pool1.

 

Build Rsync for Android Yourself

To build rsync for Android you’ll need to have the Android NDK installed already.

Then clone the rsync for android source (e.g. from CyanogenMod LineageOS) …

… create the missing
jni/Application.mk
build file (e.g. from this Gist) and adapt it to your case

… and start the build with

You’ll find your self-build rsync in
obj/local/*/rsync
. 😀

Update 2017-10-06:

  • Updated sources from CyanogenMod to LineageOS.
  • Added links to Gist and Andoid NDK docs
  • Updated steps to work with up-to-date setups

If you get something like the following warnings and errors …

… you probably need to update
config.h
and change
/* #undef MAJOR_IN_SYSMACROS */
to
#define MAJOR_IN_SYSMACROS 1
.

too long for Unix domain socket

If you’re an Ansible user and encounter the following error:

you need to set the control_path option in your ansible.cfg file to tell SSH to use shorter path names for the control socket. You should have a look at the ssh_config(5) man page  (under
ControlPath
) for a list of possible substitutions.

I chose:

Making RabbitMQ Recover from (a)Mnesia

In the company I work for we’re using RabbitMQ to offload non-timecritical processing of tasks. To be able to recover in case RabbitMQ goes down our queues are durable and all our messages are marked as persistent. We generally have a very low number of messages in flight at any moment in time. There’s just one queue with a decent amount of them: the “failed messages” dump.

The Problem

It so happens that after a botched update to the most recent version of RabbitMQ (3.5.3 at the time) our admins had to nuke the server and install it from scratch. They had made a backup of RabbitMQ’s Mnesia database and I was tasked to recover the messages from it.
This is the story of how I did it.

Since our RabbitMQ was configured to persist all the messages this should be generally possible. Surely I wouldn’t be the first one to attempt this. 😐

Looking through the Internet it seems there’s no way of ex/importing a node’s configuration if it’s not running. I couldn’t find any documentation on how to import a Mnesia backup into a new node or extract data from it into a usable form. 😞

The Idea

My idea was to setup a virtual machine (running Debian Wheezy) with RabbitMQ and then to somehow make it read/recover and run the broken server’s database.

In the following you’ll see the following placeholders:

  • RABBITMQ_MNESIA_BASE will be /var/lib/rabbitmq/mnesia  on Debian (see RabbitMQ’S file locations)
  • RABBITMQ_MNESIA_DIR is just $RABBITMQ_MNESIA_BASE/$RABBITMQ_NODENAME
  • BROKEN_NODENAME the $RABBITMQ_NODENAME of the broken server we have backups from
  • BROKEN_HOST the hostname of said server

One more thing before we start: if I say “fix permissions” below I mean

1st Try

My first try was to just copy the broken node’s Mnesia files to the VM’s $RABBITMQ_MNESIA_DIR failed. The files contained node names that RabbitMQ tried to reach but were unreachable from the VM.

So I tried to be a little bit more picky on what I copied.

First I had to reset $RABBITMQ_MNESIA_DIR by deleting it and have RabbitMQ recreate it. (I needed to do this way too many times 😭)

Stopping RabbitMQ I tried to feed it the broken server’s data in piecemeal fashion. This time I only copied the rabbit_*.[DCD,DCL]  and restarted RabbitMQ.

RabbitMQ Management Interface lists all the queues, but the node it thinks they're on is "down"
RabbitMQ’s management interface lists all the queues, but it thinks the node they’re on is “down”

Looking at the web management interface there were all the queues we were missing, but they were “down” and clicking on them told you

The object you clicked on was not found; it may have been deleted on the server.

Copying any more data didn’t solve the issue. So this was a dead end. 😫

2nd Try

So I thought why doesn’t the RabbitMQ in the VM pretend to be the exact same node as on the broken server?

So I created a /etc/rabbitmq/rabbitmq-env.conf  with NODENAME=$BROKEN_NODENAME  in there.

I copied the backup to $RABBITMQ_MNESIA_DIR (now with the new node name) and fixed the permissions.

Now starting RabbitMQ failed with

I edited /etc/hosts  to add $BROKEN_HOST to the list of names that resolve to 127.0.0.1.

Now restarting RabbitMQ failed with yet another error:

Now what? Why don’t I try to give it the Mnesia files piece by piece again?

  • Reset $RABBITMQ_MNESIA_DIR
  • Stop RabbitMQ
  • Copy rabbit_*  files in again and fix their permissions
  • Start RabbitMQ

All our queues were back and all their configuration seemed OK as well. But we still didn’t have our messages back yet.

RabbitMQ Data Recovery Screen Shot 2 - Node Up, Queues Empty
The queues have been restored, but they have no messages in them

Solution

So I tried to copy more and more files over from the backup repeating the above steps. I finally reached my goal after copying rabbit_* , msg_store_* , queues  and recovery.dets. Fixing their permissions and starting RabbitMQ it had all the queues restored with all the messages in them. 😂

RabbitMQ Data Recovery Screen Shot 3 - Messages Restored
Queues and messages restored

Now I could use ordinary methods to extract all the messages. Dumping all the messages and examining them they looked OK. Publishing the recovered messages to the new server I was pretty euphoric. 😁

Bottle Plugin Lifecycle

If you use Python‘s Bottle micro-framework there’ll be a time where you’ll want to add custom plugins. To get a better feeling on what code gets executed when, I created a minimal Bottle app with a test plugin that logs what code gets executed. I uesed it to test both global and route-specific plugins.

When Python loads the module you’ll see that the plugins’
__init__()
and
setup()
methods will be called immediately when they are installed on the app or applied to the route. This happens in the order they appear in the code. Then the app is started.

The first time a route is called Bottle executes the plugins’
apply()
methods. This happens in “reversed order” of installation (which makes sense for a nested callback chain). This means first the route-specific plugins get applied then the global ones. Their result is cached, i.e. only the inner/wrapped function is executed from here on out.

Then for every request the
apply()
method’s inner function is executed. This happens in the “original” order again.

Below you can see the code and example logs for two requests. You can also clone the Gist and do your own experiments.

Looking Up Crash Reports In OS X

If you find yourself–like me–in the situation that your Mac has crashed and you want to retrieve the crash reports (some call them logs 😉 )? Well, there are basically two ways.

You can look them up with the “Console” tool (find it in  /Applications/Utilities/Console  or with Spotlight). Open the “System Diagnostic Reports” section on the left and find an entry similar to  Kernel_<date>_<your_pc_name>.panic  at the top.

You can also find these reports as text files under  /Library/Logs/DiagnosticReports with the same names. OS X will open them with the Console tool per default.

Cheers. 😀

Unsafe Chrome Sometimes Necessary

In my work – every now and then – I found myself in need of a browser with reduced security checks (mainly to gloss over cross domain XMLHttpRequests and SSL certificate violations) for testing purposes. I didn’t want to take the risk and use my main browser session with these settings, so I made me a script (also available as a Gist). 🙂

Tip:
If you use oh my ZSH you can save this file in ~/.oh-my-zsh/custom/plugins/chrome-unsafe/chrome-unsafe.plugin.zsh and add “chrome-unsafe” to your list of used plugins in ~/.zshrc