Routing My Way Out With IPv6: NPT6

This article is part of a series of how I built a WireGuard tunnel for getting IPv6 connectivity. Where the last step was to figure out how to route packets from devices in my private network through the WireGuard tunnel to the Internet.

I’ve explored three different methods for solving this:

I’ll try to show how to set each of them up and try to convey their pros and cons.

TL;DR

You should always consider IPv6-PD first!

Consider any other option only if:

  • you have a “weird” setup or want to support an esoteric use case (like I do e.g. with too many local subnets for too long a public prefix)
  • you’re willing to set up, debug and maintain a somewhat experimental configuration
  • you more or less understand the tradeoffs
  • all of the above!

Starting Point

I’ll assume the following has been set up:

  • default OpenWRT networks named “LAN”, “WAN”, “WAN6”
  • default OpenWRT firewall rules
  • an ULA prefix of fd00:11:22::/48
  • an IPv6 WireGuard tunnel with the endpoint on our OpenWRT router being 2000:30:40:50::2
  • the remote WireGurad tunnel end point forwards the whole 2000:30:40:50::/64 to our OpenWRT router

NPTv6 (Network Prefix Translation)

This is probably the least publicly documented method of all. Discussions and tutorials are scarce. Its use cases are esoteric and probably better solved in other ways. But it’s the most interesting method, because it’s conceptually even simpler than NAT6, but only viable with IPv6 addresses.

NPT basically means that you swap the prefix part of an IPv6 address with another same-sized prefix. It exploits two facts about IPv6 addresses. The first one is that prefixes can be at most 64 bits long (i.e. for a /64) leaving the interface identifier (i.e. the second half of the IPv6 address) untouched. The second one is that interface identifiers are basically random (i.e. because they’re either derived from (globally) unique MAC addresses or they’re randomly generated temporary addresses) and hence won’t clash. This allows for stateless, NAT-like behavior (i.e.without the “expensive” tracking of NATed connections).

You can configure NPT to be bidirectional which maps prefixes in both directions basically creating a 1:1 mapping. If you’re doing this you’re probably better off just announcing multiple prefixes to your devices or creating custom routes to bridge two networks.

An even more esoteric use case is when you create one or more unidirectional mappings allowing you to multiplex multiple networks onto one. This works great, because the interface identifiers are basically random and can be left as they are. In my tests having one-way mappings still managed to route the responses correctly although strictly speaking it shouldn’t. 🤨 I suspect that this worked accidentally, because of the standard firewall “conntrack” (i.e. connection tracking) rules. 🤔

Setup

On the “Network > Interfaces” page edit the “WAN6” interface and set “Protocol” to “unmanaged”. And make sure the “WAN6_WG” addresses say 2000:30:40:50::2/64 (note the /64 at the end).

Similar to the NAT6 case we need a custom firewall script. You have to install the iptables-mod-nat-extra package. I’ve created a Gist for the script. Save it to /etc/firewall.npt6 and instruct the firewall to run it when being reloaded by adding the following section to /etc/config/firewall:

config include 'npt6'
        option path '/etc/firewall.npt6'
        option reload '1'

After restarting the firewall with /etc/init.d/firewall restart you should be good to go.

As described at the top of the firewall script you can configure mappings by adding npt6 config sections to /etc/config/firewall (sorry, there’s no UI for this 😅).

config npt6
        option src_interface 'lan'
        option dest_interface 'wan6_wg'
        option output '1'

This is the minimal setup. Just add more sections for more source and destination network pairs. Run /etc/init.d/firewall reload to apply new configurations.

In my tests all devices could connect to IPv6 services on the internet without problems. But devices always preferred IPv4 connections over IPv6 ones. This was tricky to solve, but it comes down to this:

When a domain has both public/global IPv4 and IPv6 addresses your devices tries to determine how to connect to it. It’ll generally prefer IPv6 over IPv4, but actually its more complicated than that. All IPv4 addresses are treated as global during address selection while IPv6 addresses are classified differently depending on the prefix. It just so happens that from the outside it looks something like this: global IPv6 address > IPv4 addresses > IPv6 ULAs. It’s a little more complicated

Since we don’t have a global IPv6 address, IPv4 is preferred assuming that private IPv4 addresses will generally be NATed to the Internet while ULA prefixes won’t. 😞

This was tricky to solve. All related questions on the Internet revolved around how to prefer IPv4 over IPv6, but the solution was not invertible. It boils down to changing /etc/gai.conf to classify your ULA prefix the same as a global ones. You can accomplish this by adding a label line for your ULA (i.e. fd00:11:22::/48 here) and giving it the same label (i.e. the last number on the line) as the line with ::/0 (i.e. 1 here for me). Finding this out took me a week of trial and error until I resigned to doing the address selection algorithm by hand. 😅

I had to uncomment all the label configuration lines and then add my custom line, because once you add a custom rule all the default ones will be reset. So to add a rule on top of the default ones I ended up with the following (note that I only added the last line, all others were part of Ubuntu’s default configuration):

...
label ::1/128       0
label ::/0          1
label 2002::/16     2
label ::/96         3
label ::ffff:0:0/96 4
label fec0::/10     5
label fc00::/7      6
label 2001:0::/32   7
label fd00:11:22::/48 1
...

I only added my network’s ULA to preserve the default behavior as much as possible and only make an exception for my network specifically. so this will change the behavior only when the device has addresses from this specific ULA.

You have to restart applications for them to pick up changes to /etc/gai.conf.

Pros

  • multiple internal networks can be multiplexed onto one upstream network (even when the upstream prefix is too long (e.g. for IPv6-PD))
  • internal devices are not directly reachable from the Internet (with unidirectional mapping) (this is not a replacement for a firewall!)

Cons

  • very little documentation and online resources
  • for your devices to use IPv6 by default you have to muck with address selection preferences on each and every one of them
  • it doesn’t fall back to IPv4 when the IPv6 tunnel goes down

List all processes and how much swap they use

Sometimes I see that my computer uses a lot of swap space, but none of the system monitors (e.g. ps, Gnome’s System Monitor, top, btop) show which processes are to blame. There’re several memory metrics (e.g. total swap usage), but never per process swap usage. 😠

There’s a StackExchange question that asks the same thing, but the solutions are all not well scripted, slow or “meh” for other reasons … so I tackled the problem myself. 😝

All the information is basically buried in the /proc/*/status files. Among other things they contain the amount of swap a process uses, its PID and even better: its name. So we have to go through all the files, grep the useful lines, extract the data from those lines and recombine them.

I tried different combinations of grep and sed and Bash string interpolation … and while it worked, it was even slower than the StackExchange suggestions. 😯😅😞 This looked more and more like a “if I knew grep/sed/awk better I wouldn’t need to invoke 4 sub shells/pipes/processes for each file” kind of problem. I remembered Bryan Cantrill making an offhanded remark once that awk had a simple and concise language and a great manual.

If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk.

Bryan Cantrill

I had put off diving in for too long. So I started reading and roughly 20 minutes in I knew enough to solve the whole problem (almost) entirely in awk (it still needs the shell for filename globbing 😕🤷). And … it’s blazingly fast! And … you can also combine it with sort and head to quickly find the worst offenders. 😎

You can find the code in this Gist. There’s a simpler version in an earlier revision. 😉

Scaffolding for fetching and parsing emails from IMAP with Python

A friend asked me if I could help him write a Python script for fetching and processing data from emails in his mailbox … Well, the thing with emails is that they’re a pain to work with (in any form). So, I tried to help him out with a little scaffolding (also available as a Gist).

Backup And Restore Your Android Phone With ADB (And rsync)

Based on my previous scripts and inspired by two blog posts that I stumbled upon I tackled the “backup all my apps, settings and data” problem for my Android devices again. The “new” solutions both use

rsync

  instead of

adb pull

  for file transfers. They both use ADB to start a rsync daemon on the device, forward its ports to localhost and run rsync against it from your host.

Simon’s solution assumes your phone has rsync already (e.g. because you run CyanogenMod) and can become root via

adb root

. It clones all files from the phone (minus

/dev

 ,

/sys

 ,

/proc

  etc.). He also configures udev to start the backup automatically when the phone is plugged in.

pts solves the setup without necessarily becoming root. He also has a way of providing a rsync binary to phones that don’t have any (e.g. when running OxygenOS). He also has a few tricks on how to debug the rsync daemon setup on the phone.

I’ve tried to combine both methods. My approach doesn’t require adb or rsync to be run as root. It’ll use the the system’s rsync when available or temporarily upload and use a backup one extracted from Cyanogen OS (for my OnePlus One). Android won’t allow you to 

chmod +x

a file uploaded to

/sdcard

, but in

/data/local/tmp

it works. ?

The scripts will currently only backup and restore all of your 

/sdcard

directory. Assuming you’re also using something like Titanium Backup you’ll be able to backup and restore all your apps, settings and data. To reduce the amount of data to copy it uses rsync filters to exclude caches and other files that you definitely don’t want synced (

.DS_Store

  files anyone?).

At the moment there’s one caveat: I had to disable restoring modification times (i.e. use

--no-times

 ) because of an obnoxious error (they will be backuped fine, only restoring is the problem): ?

mkstemp “…” (in root) failed: Operation not permitted (1)

Additionally if you’re on the paranoid side you can also build your own rsync for Android to use as the backup binary.

The code and a ton of documentation can be found on GitHub. Comments and suggestions are welcome. ?

Build Rsync for Android Yourself

To build rsync for Android you’ll need to have the Android NDK installed already.

Then clone the rsync for android source (e.g. from CyanogenMod LineageOS) …

git clone https://github.com/LineageOS/android_external_rsync.git
cd android_external_rsync
# checkout the most recent branch
git checkout cm-14.1

… create the missing

jni/Application.mk

build file (e.g. from this Gist) and adapt it to your case

… and start the build with

export NDK_PROJECT_PATH=pwd ndk-build -d rsync

You’ll find your self-build rsync in

obj/local/*/rsync

. ?

Update 2017-10-06:

  • Updated sources from CyanogenMod to LineageOS.
  • Added links to Gist and Andoid NDK docs
  • Updated steps to work with up-to-date setups

If you get something like the following warnings and errors …

[...]
./flist.c:454:16: warning: implicit declaration of function 'major' is invalid in C99
      [-Wimplicit-function-declaration]
                        if ((uint32)major(rdev) == rdev_major)
                                    ^
./flist.c:458:41: warning: implicit declaration of function 'minor' is invalid in C99
      [-Wimplicit-function-declaration]
                        if (protocol_version < 30 && (uint32)minor(rdev) <= 0xFFu)
                                                             ^
./flist.c:467:11: warning: implicit declaration of function 'makedev' is invalid in C99
      [-Wimplicit-function-declaration]
                        rdev = MAKEDEV(major(rdev), 0);
                               ^
./rsync.h:446:36: note: expanded from macro 'MAKEDEV'
#define MAKEDEV(devmajor,devminor) makedev(devmajor,devminor)
                                   ^
3 warnings generated.
[...]
./flist.c:473: error: undefined reference to 'makedev'
./flist.c:454: error: undefined reference to 'major'
./flist.c:457: error: undefined reference to 'major'
./flist.c:458: error: undefined reference to 'minor'
./flist.c:467: error: undefined reference to 'major'
./flist.c:467: error: undefined reference to 'makedev'
./flist.c:617: error: undefined reference to 'major'
./flist.c:619: error: undefined reference to 'minor'
./flist.c:621: error: undefined reference to 'minor'
./flist.c:788: error: undefined reference to 'makedev'
./flist.c:869: error: undefined reference to 'makedev'
./flist.c:1027: error: undefined reference to 'minor'
clang++: error: linker command failed with exit code 1 (use -v to see invocation) 
make: *** [obj/local/armeabi-v7a/rsync] Error 1

… you probably need to update

config.h

and change

/* #undef MAJOR_IN_SYSMACROS */

to

#define MAJOR_IN_SYSMACROS 1

.

CFSSL FTW

After reading how CloudFlare handles their PKI and that LetsEncrypt will use it I wanted to give CFSSL a shot.

Reading the project’s documentation doesn’t really help in building your own CA, but searching the Internet I found Fernando Barillas’ blog explaining how to create your own root certificate and how to create intermediate certificates from this.

I took it a step further I wrote a script generating new certificates for several services with different intermediates and possibly different configurations (e.g. depending on your distro and services certain cyphers (e.g. using ECC) may not be supported).
I also streamlined generating service specific key, cert and chain files. 😀

Have a look at the full Gist or just the most interesting part:

You’ll still have to deploy them yourself.

Update 2016-10-04:
Fixed some issues with this Gist.

  • Fixed a bug where intermediate CA certificates weren’t marked as CAs any more
  • Updated the example CSRs and the script so it can now be run without errors

Update 2017-10-08:

  • Cleaned up `renew-certs.sh` by extracting functions for generating root CA, intermediate CA and service keys.

A Service Monitor built with Polymer

I tried to build a service monitor having the following features:

  • showing the reachability of HTTP servers
  • plotting the amount of messages in a specific RabbitMQ queue
  • plotting the amount of queues with specific prefixes
  • showing the status of RabbitMQ queues i.e. how many messages are in there? are there any consumers? are they hung?
  • showing the availability of certain Redis clients

Well, you can find the result on GitHub.
It uses two things I published before: polymer-flot and flot-sparklines. 😀

An example dashboard:

polymer-service-monitor screen shot