Moving LXD Containers From One Pool to Another

When I started playing with LXD I just accepted the default storage configuration which creates an image file and uses that to initialize a ZFS pool. Since I’m using ZFS as my main file system this seemed silly as LXD can use an existing dataset as a source for a storage pool. So I wanted to migrate my existing containers to the new storage pool.

Although others seemed to to have the same problem there was no ready answer. Digging through the documentation I finally found out that the lxc move  command had a -s  option … I had an idea. ? Here’s what I came up with …

Preparation

First we create the dataset on the existing ZFS pool and add it to LXC.

sudo zfs create -o mountpoint=none mypool/lxd
lxc storage create pool2 zfs source=mypool/lxd

lxc storage list should show something like this now:

+-------+-------------+--------+--------------------+---------+
| NAME  | DESCRIPTION | DRIVER |       SOURCE       | USED BY |
+-------+-------------+--------+--------------------+---------+
| pool1 |             | zfs    | /path/to/pool1.img | 2       |
+-------+-------------+--------+--------------------+---------+
| pool2 |             | zfs    | mypool/lxd         | 0       |
+-------+-------------+--------+--------------------+---------+

pool1 is the old pool backed by the image file and is used by some containers at the moment as can be seen in the “Used By” column. pool2 is added by not used by any contaiers yet.

Moving

We now try to move our containers to pool2.

# move container to pool2
lxc move some_container some_container-moved -s=pool2
# rename container back for sanity ;)
lxc move some_container-moved some_container

We can check with lxc storage list whether we succeeded.

+-------+-------------+--------+--------------------+---------+
| NAME  | DESCRIPTION | DRIVER |       SOURCE       | USED BY |
+-------+-------------+--------+--------------------+---------+
| pool1 |             | zfs    | /path/to/pool1.img | 1       |
+-------+-------------+--------+--------------------+---------+
| pool2 |             | zfs    | mypool/lxd         | 1       |
+-------+-------------+--------+--------------------+---------+

Indeed pool2 is beeing used now. ? Just to be sure we check that zfs list -r mypool/lxd  also reflects this.

NAME                                  USED  AVAIL  REFER  MOUNTPOINT
mypool/lxd/containers                 1,08G  92,9G    24K  none
mypool/lxd/containers/some_container  1,08G  92,9G   704M  /var/snap/lxd/common/lxd/storage-pools/pool2/containers/some_container
mypool/lxd/custom                       24K  92,9G    24K  none
mypool/lxd/deleted                      24K  92,9G    24K  none
mypool/lxd/images                       24K  92,9G    24K  none
mypool/lxd/snapshots                    24K  92,9G    24K  none

Awesome!

⚠ Note that this only moves the container, but not the LXC image it was cloned off of.

We can repeat this until all containers we care about are moved over to pool2.

Cleanup

To prevent new containers to use pool1  we have to edit the default  profile.

# change devices.root.pool to pool2
lxc profile edit default

Finally …. when we’re happy with the migration and we’ve verified that everything works as expected we can now remove pool1.

lxc storage rm pool1

 

Build Rsync for Android Yourself

To build rsync for Android you’ll need to have the Android NDK installed already.

Then clone the rsync for android source (e.g. from CyanogenMod LineageOS) …

git clone https://github.com/LineageOS/android_external_rsync.git
cd android_external_rsync
# checkout the most recent branch
git checkout cm-14.1

… create the missing

jni/Application.mk

build file (e.g. from this Gist) and adapt it to your case

… and start the build with

export NDK_PROJECT_PATH=pwd ndk-build -d rsync

You’ll find your self-build rsync in

obj/local/*/rsync

. ?

Update 2017-10-06:

  • Updated sources from CyanogenMod to LineageOS.
  • Added links to Gist and Andoid NDK docs
  • Updated steps to work with up-to-date setups

If you get something like the following warnings and errors …

[...]
./flist.c:454:16: warning: implicit declaration of function 'major' is invalid in C99
      [-Wimplicit-function-declaration]
                        if ((uint32)major(rdev) == rdev_major)
                                    ^
./flist.c:458:41: warning: implicit declaration of function 'minor' is invalid in C99
      [-Wimplicit-function-declaration]
                        if (protocol_version < 30 && (uint32)minor(rdev) <= 0xFFu)
                                                             ^
./flist.c:467:11: warning: implicit declaration of function 'makedev' is invalid in C99
      [-Wimplicit-function-declaration]
                        rdev = MAKEDEV(major(rdev), 0);
                               ^
./rsync.h:446:36: note: expanded from macro 'MAKEDEV'
#define MAKEDEV(devmajor,devminor) makedev(devmajor,devminor)
                                   ^
3 warnings generated.
[...]
./flist.c:473: error: undefined reference to 'makedev'
./flist.c:454: error: undefined reference to 'major'
./flist.c:457: error: undefined reference to 'major'
./flist.c:458: error: undefined reference to 'minor'
./flist.c:467: error: undefined reference to 'major'
./flist.c:467: error: undefined reference to 'makedev'
./flist.c:617: error: undefined reference to 'major'
./flist.c:619: error: undefined reference to 'minor'
./flist.c:621: error: undefined reference to 'minor'
./flist.c:788: error: undefined reference to 'makedev'
./flist.c:869: error: undefined reference to 'makedev'
./flist.c:1027: error: undefined reference to 'minor'
clang++: error: linker command failed with exit code 1 (use -v to see invocation) 
make: *** [obj/local/armeabi-v7a/rsync] Error 1

… you probably need to update

config.h

and change

/* #undef MAJOR_IN_SYSMACROS */

to

#define MAJOR_IN_SYSMACROS 1

.

too long for Unix domain socket

If you’re an Ansible user and encounter the following error:

unix_listener: "..." too long for Unix domain socket

you need to set the control_path option in your ansible.cfg file to tell SSH to use shorter path names for the control socket. You should have a look at the ssh_config(5) man page  (under

ControlPath

) for a list of possible substitutions.

I chose:

control_path = %(directory)s/ssh-%%C

Making RabbitMQ Recover from (a)Mnesia

In the company I work for we’re using RabbitMQ to offload non-timecritical processing of tasks. To be able to recover in case RabbitMQ goes down our queues are durable and all our messages are marked as persistent. We generally have a very low number of messages in flight at any moment in time. There’s just one queue with a decent amount of them: the “failed messages” dump.

The Problem

It so happens that after a botched update to the most recent version of RabbitMQ (3.5.3 at the time) our admins had to nuke the server and install it from scratch. They had made a backup of RabbitMQ’s Mnesia database and I was tasked to recover the messages from it.
This is the story of how I did it.

Since our RabbitMQ was configured to persist all the messages this should be generally possible. Surely I wouldn’t be the first one to attempt this. ?

Looking through the Internet it seems there’s no way of ex/importing a node’s configuration if it’s not running. I couldn’t find any documentation on how to import a Mnesia backup into a new node or extract data from it into a usable form. ?

The Idea

My idea was to setup a virtual machine (running Debian Wheezy) with RabbitMQ and then to somehow make it read/recover and run the broken server’s database.

In the following you’ll see the following placeholders:

  • RABBITMQ_MNESIA_BASE will be
    /var/lib/rabbitmq/mnesia

      on Debian (see RabbitMQ’S file locations)

  • RABBITMQ_MNESIA_DIR is just $RABBITMQ_MNESIA_BASE/$RABBITMQ_NODENAME
  • BROKEN_NODENAME the $RABBITMQ_NODENAME of the broken server we have backups from
  • BROKEN_HOST the hostname of said server

One more thing before we start: if I say “fix permissions” below I mean

sudo chown -R rabbitmq:rabbitmq $RABBITMQ_MNESIA_DIR

1st Try

My first try was to just copy the broken node’s Mnesia files to the VM’s $RABBITMQ_MNESIA_DIR failed. The files contained node names that RabbitMQ tried to reach but were unreachable from the VM.

Error description:
   {could_not_start,rabbit,
       {{failed_to_cluster_with,
            ['$BROKEN_NODENAME'],
            "Mnesia could not connect to any nodes."},
        {rabbit,start,[normal,[]]}}}

So I tried to be a little bit more picky on what I copied.

First I had to reset $RABBITMQ_MNESIA_DIR by deleting it and have RabbitMQ recreate it. (I needed to do this way too many times ?)

sudo service rabbitmq-server stop
rm -r $RABBITMQ_MNESIA_DIR
sudo service rabbitmq-server start

Stopping RabbitMQ I tried to feed it the broken server’s data in piecemeal fashion. This time I only copied the

rabbit_*.[DCD,DCL]

  and restarted RabbitMQ.

RabbitMQ Management Interface lists all the queues, but the node it thinks they're on is "down"
RabbitMQ’s management interface lists all the queues, but it thinks the node they’re on is “down”

Looking at the web management interface there were all the queues we were missing, but they were “down” and clicking on them told you

The object you clicked on was not found; it may have been deleted on the server.

Copying any more data didn’t solve the issue. So this was a dead end. ?

2nd Try

So I thought why doesn’t the RabbitMQ in the VM pretend to be the exact same node as on the broken server?

So I created a

/etc/rabbitmq/rabbitmq-env.conf

  with

NODENAME=$BROKEN_NODENAME

  in there.

I copied the backup to $RABBITMQ_MNESIA_DIR (now with the new node name) and fixed the permissions.

Now starting RabbitMQ failed with

ERROR: epmd error for host $BROKEN_HOST: nxdomain (non-existing domain)

I edited

/etc/hosts

  to add $BROKEN_HOST to the list of names that resolve to 127.0.0.1.

Now restarting RabbitMQ failed with yet another error:

Error description:
   {could_not_start,rabbit,
       {{schema_integrity_check_failed,
            [{table_attributes_mismatch,rabbit_queue,
                 [name,durable,auto_delete,exclusive_owner,arguments,pid,
                  slave_pids,sync_slave_pids,recoverable_slaves,policy,
                  gm_pids,decorators,state],
                 [name,durable,auto_delete,exclusive_owner,arguments,pid,
                  slave_pids,sync_slave_pids,mirror_nodes,policy]}]},
        {rabbit,start,[normal,[]]}}}

Now what? Why don’t I try to give it the Mnesia files piece by piece again?

  • Reset $RABBITMQ_MNESIA_DIR
  • Stop RabbitMQ
  • Copy
    rabbit_*

      files in again and fix their permissions

  • Start RabbitMQ

All our queues were back and all their configuration seemed OK as well. But we still didn’t have our messages back yet.

RabbitMQ Data Recovery Screen Shot 2 - Node Up, Queues Empty
The queues have been restored, but they have no messages in them

Solution

So I tried to copy more and more files over from the backup repeating the above steps. I finally reached my goal after copying

rabbit_*

 ,

msg_store_*

 ,

queues

  and

recovery.dets

. Fixing their permissions and starting RabbitMQ it had all the queues restored with all the messages in them. ?

RabbitMQ Data Recovery Screen Shot 3 - Messages Restored
Queues and messages restored

Now I could use ordinary methods to extract all the messages. Dumping all the messages and examining them they looked OK. Publishing the recovered messages to the new server I was pretty euphoric. ?

Bottle Plugin Lifecycle

If you use Python‘s Bottle micro-framework there’ll be a time where you’ll want to add custom plugins. To get a better feeling on what code gets executed when, I created a minimal Bottle app with a test plugin that logs what code gets executed. I uesed it to test both global and route-specific plugins.

When Python loads the module you’ll see that the plugins’

__init__()

and

setup()

methods will be called immediately when they are installed on the app or applied to the route. This happens in the order they appear in the code. Then the app is started.

The first time a route is called Bottle executes the plugins’

apply()

methods. This happens in “reversed order” of installation (which makes sense for a nested callback chain). This means first the route-specific plugins get applied then the global ones. Their result is cached, i.e. only the inner/wrapped function is executed from here on out.

Then for every request the

apply()

method’s inner function is executed. This happens in the “original” order again.

Below you can see the code and example logs for two requests. You can also clone the Gist and do your own experiments.

Looking Up Crash Reports In OS X

If you find yourself–like me–in the situation that your Mac has crashed and you want to retrieve the crash reports (some call them logs 😉 )? Well, there are basically two ways.

You can look them up with the “Console” tool (find it in 

/Applications/Utilities/Console

  or with Spotlight). Open the “System Diagnostic Reports” section on the left and find an entry similar to 

Kernel_<date>_<your_pc_name>.panic

  at the top.

You can also find these reports as text files under 

/Library/Logs/DiagnosticReports

with the same names. OS X will open them with the Console tool per default.

Cheers. 😀

Unsafe Chrome Sometimes Necessary

In my work – every now and then – I found myself in need of a browser with reduced security checks (mainly to gloss over cross domain XMLHttpRequests and SSL certificate violations) for testing purposes. I didn’t want to take the risk and use my main browser session with these settings, so I made me a script (also available as a Gist). 🙂

Tip:
If you use oh my ZSH you can save this file in

~/.oh-my-zsh/custom/plugins/chrome-unsafe/chrome-unsafe.plugin.zsh

and add “chrome-unsafe” to your list of used plugins in

~/.zshrc

 

Fixing Bioshock’s Language Settings

I got myself Bioshock 1 and 2 during Steam’s Summer Sale and just came around to play it.
After installation I found out that the game was in German (while my Windows and Steam are in English)!?!? … I hate dubbed films and games!

Sadly Bioshiock doesn’t allow you to change the language in the settings menu so I had to look around. After piecing together information from several 5+ year old, outdated forum posts I found the solution.

  1. Close the game
  2. Open C:\Users\<Username>\AppData\Roaming\Bioshock\bioshock.ini
  3. Find the
    [Engine.Engine]

      section

  4. Set
    Language=int

      (“int” is for international i.e. English)

  5. Save the file and restart the game

Everything should be in English now. 😀