Tag: Source Code

Routing My Way Out With IPv6: NPT6

This article is part of a series of how I built a WireGuard tunnel for getting IPv6 connectivity. Where the last step was to figure out how to route packets from devices in my private network through the WireGuard tunnel to the Internet.

I’ve explored three different methods for solving this:

I’ll try to show how to set each of them up and try to convey their pros and cons.

TL;DR

You should always consider IPv6-PD first!

Consider any other option only if:

  • you have a “weird” setup or want to support an esoteric use case (like I do e.g. with too many local subnets for too long a public prefix)
  • you’re willing to set up, debug and maintain a somewhat experimental configuration
  • you more or less understand the tradeoffs
  • all of the above!

Starting Point

I’ll assume the following has been set up:

  • default OpenWRT networks named “LAN”, “WAN”, “WAN6”
  • default OpenWRT firewall rules
  • an ULA prefix of fd00:11:22::/48
  • an IPv6 WireGuard tunnel with the endpoint on our OpenWRT router being 2000:30:40:50::2
  • the remote WireGurad tunnel end point forwards the whole 2000:30:40:50::/64 to our OpenWRT router

NPTv6 (Network Prefix Translation)

This is probably the least publicly documented method of all. Discussions and tutorials are scarce. Its use cases are esoteric and probably better solved in other ways. But it’s the most interesting method, because it’s conceptually even simpler than NAT6, but only viable with IPv6 addresses.

NPT basically means that you swap the prefix part of an IPv6 address with another same-sized prefix. It exploits two facts about IPv6 addresses. The first one is that prefixes can be at most 64 bits long (i.e. for a /64) leaving the interface identifier (i.e. the second half of the IPv6 address) untouched. The second one is that interface identifiers are basically random (i.e. because they’re either derived from (globally) unique MAC addresses or they’re randomly generated temporary addresses) and hence won’t clash. This allows for stateless, NAT-like behavior (i.e.without the “expensive” tracking of NATed connections).

You can configure NPT to be bidirectional which maps prefixes in both directions basically creating a 1:1 mapping. If you’re doing this you’re probably better off just announcing multiple prefixes to your devices or creating custom routes to bridge two networks.

An even more esoteric use case is when you create one or more unidirectional mappings allowing you to multiplex multiple networks onto one. This works great, because the interface identifiers are basically random and can be left as they are. In my tests having one-way mappings still managed to route the responses correctly although strictly speaking it shouldn’t. 🤨 I suspect that this worked accidentally, because of the standard firewall “conntrack” (i.e. connection tracking) rules. 🤔

Setup

On the “Network > Interfaces” page edit the “WAN6” interface and set “Protocol” to “unmanaged”. And make sure the “WAN6_WG” addresses say 2000:30:40:50::2/64 (note the /64 at the end).

Similar to the NAT6 case we need a custom firewall script. You have to install the iptables-mod-nat-extra package. I’ve created a Gist for the script. Save it to /etc/firewall.npt6 and instruct the firewall to run it when being reloaded by adding the following section to /etc/config/firewall:

config include 'npt6'
        option path '/etc/firewall.npt6'
        option reload '1'

After restarting the firewall with /etc/init.d/firewall restart you should be good to go.

As described at the top of the firewall script you can configure mappings by adding npt6 config sections to /etc/config/firewall (sorry, there’s no UI for this 😅).

config npt6
        option src_interface 'lan'
        option dest_interface 'wan6_wg'
        option output '1'

This is the minimal setup. Just add more sections for more source and destination network pairs. Run /etc/init.d/firewall reload to apply new configurations.

In my tests all devices could connect to IPv6 services on the internet without problems. But devices always preferred IPv4 connections over IPv6 ones. This was tricky to solve, but it comes down to this:

When a domain has both public/global IPv4 and IPv6 addresses your devices tries to determine how to connect to it. It’ll generally prefer IPv6 over IPv4, but actually its more complicated than that. All IPv4 addresses are treated as global during address selection while IPv6 addresses are classified differently depending on the prefix. It just so happens that from the outside it looks something like this: global IPv6 address > IPv4 addresses > IPv6 ULAs. It’s a little more complicated

Since we don’t have a global IPv6 address, IPv4 is preferred assuming that private IPv4 addresses will generally be NATed to the Internet while ULA prefixes won’t. 😞

This was tricky to solve. All related questions on the Internet revolved around how to prefer IPv4 over IPv6, but the solution was not invertible. It boils down to changing /etc/gai.conf to classify your ULA prefix the same as a global ones. You can accomplish this by adding a label line for your ULA (i.e. fd00:11:22::/48 here) and giving it the same label (i.e. the last number on the line) as the line with ::/0 (i.e. 1 here for me). Finding this out took me a week of trial and error until I resigned to doing the address selection algorithm by hand. 😅

I had to uncomment all the label configuration lines and then add my custom line, because once you add a custom rule all the default ones will be reset. So to add a rule on top of the default ones I ended up with the following (note that I only added the last line, all others were part of Ubuntu’s default configuration):

...
label ::1/128       0
label ::/0          1
label 2002::/16     2
label ::/96         3
label ::ffff:0:0/96 4
label fec0::/10     5
label fc00::/7      6
label 2001:0::/32   7
label fd00:11:22::/48 1
...

I only added my network’s ULA to preserve the default behavior as much as possible and only make an exception for my network specifically. so this will change the behavior only when the device has addresses from this specific ULA.

You have to restart applications for them to pick up changes to /etc/gai.conf.

Pros

  • multiple internal networks can be multiplexed onto one upstream network (even when the upstream prefix is too long (e.g. for IPv6-PD))
  • internal devices are not directly reachable from the Internet (with unidirectional mapping) (this is not a replacement for a firewall!)

Cons

  • very little documentation and online resources
  • for your devices to use IPv6 by default you have to muck with address selection preferences on each and every one of them
  • it doesn’t fall back to IPv4 when the IPv6 tunnel goes down

List all processes and how much swap they use

Sometimes I see that my computer uses a lot of swap space, but none of the system monitors (e.g. ps, Gnome’s System Monitor, top, btop) show which processes are to blame. There’re several memory metrics (e.g. total swap usage), but never per process swap usage. 😠

There’s a StackExchange question that asks the same thing, but the solutions are all not well scripted, slow or “meh” for other reasons … so I tackled the problem myself. 😝

All the information is basically buried in the /proc/*/status files. Among other things they contain the amount of swap a process uses, its PID and even better: its name. So we have to go through all the files, grep the useful lines, extract the data from those lines and recombine them.

I tried different combinations of grep and sed and Bash string interpolation … and while it worked, it was even slower than the StackExchange suggestions. 😯😅😞 This looked more and more like a “if I knew grep/sed/awk better I wouldn’t need to invoke 4 sub shells/pipes/processes for each file” kind of problem. I remembered Bryan Cantrill making an offhanded remark once that awk had a simple and concise language and a great manual.

If you get the awk programming language manual…you’ll read it in about two hours and then you’re done. That’s it. You know all of awk.

Bryan Cantrill

I had put off diving in for too long. So I started reading and roughly 20 minutes in I knew enough to solve the whole problem (almost) entirely in awk (it still needs the shell for filename globbing 😕🤷). And … it’s blazingly fast! And … you can also combine it with sort and head to quickly find the worst offenders. 😎

You can find the code in this Gist. There’s a simpler version in an earlier revision. 😉

Skip unreadable files in awk scripts

Have you ever needed to give awk multiple files to process, but wanted it to skip files that it can’t open? Use this snippet!

BEGINFILE {
    # skip files that cannot be opened
    if ( ERRNO != "" ) {
        nextfile
    }
}

This trick is mentioned in the documentation for the BEGINFILE pattern, but not spelled out as code. You’re welcome. 😀

Moving Passwords From Firefox (Lockwise) to KeePassXC

Update 2021-03-27: Good news, it seems KeepassXC 2.7.0 learned to import Firefox timestamps. 😀

Go to about:logins in Firefox.

Click on the three dots in the top right corner, select “Export Logins…” and save your passwords info a CSV file (e.g. exported_firefox_logins.csv).

There’s one complication: Firefox will save dates as Unix timestamps (with millisecond resolution) which KeePassXC doesn’t understand, so we’ll have to help it out.

Save the following script into a file (e.g. firefox_csv_date_fix.py).

#!/usr/bin/env python3
#
# Author: Riyad Preukschas <riyad@informatik.uni-bremen.de>
# License: Mozilla Public License 2.0
# SPDX-License-Identifier: MPL-2.0


import csv
import sys

from datetime import datetime


def main():
    if len(sys.argv) != 2:
        print("Usage: {} <exported_firefox_logins.csv>".format(sys.argv[0]), file=sys.stderr)
        exit(1)

    csv_file_name = sys.argv[1]

    with open(csv_file_name, 'rt') as f:
        # field names will be determined from first row
        csv_reader = csv.DictReader(f)
        # read all rows in one go
        rows = list(csv_reader)

    # add new columns with Iso-formatted dates
    for row in rows:
        row['timeCreatedIso'] = datetime.fromtimestamp(int(row['timeCreated'])/1000).isoformat()
        row['timeLastUsedIso'] = datetime.fromtimestamp(int(row['timeLastUsed'])/1000).isoformat()
        row['timePasswordChangedIso'] = datetime.fromtimestamp(int(row['timePasswordChanged'])/1000).isoformat()

    # write out the updated rows
    csv_writer = csv.DictWriter(sys.stdout, fieldnames=rows[0].keys())
    csv_writer.writeheader()
    csv_writer.writerows(rows)


if __name__ == '__main__':
    main()

Call it giving it the path of the exported_firefox_logins.csv file and redirect the output into a new file:

python3 ./firefox_csv_date_fix.py ./exported_firefox_logins.csv > ./fixed_firefox_logins.csv

It will add new columns (named: old column name + “Iso”) with the dates converted to the ISO 8601 format KeePassXC wants.

We can now use the fixed_firefox_logins.csv file to import the logins into KeePassXC.

Select Database -> Import -> CSV File... from the menu.

It will ask you to create a new database. Just do it you can get the data into your default database later (e.g. call it firefox_logins.kdbx for now).

In the import form select “First line has field names” and match up KeePassXC’s fields with the columns from the CSV:

KeePassXC FieldFirefox CSV ColumnNotes
GroupNot Present
TitleurlFirefox doesn’t have a title field so we use the URL for it.
Usernameusername
Passwordpassword
URLurl
NoteshttpRealmI’m not sure how KeePassXC deals with HTTP authentication. There’s no special field for it, so I decided to keep it in the Notes field.
TOTPNot Present
IconNot Present
Last ModifiedtimePasswordChangedIsothe “Iso” part is important!
CreatedtimeCreatedIsothe “Iso” part is important!

Have a look at the preview at the bottom. Does the data look sane? 🤨

Sadly KeePassXC doesn’t let us import Firefox’s timeLastUsedIso field into its Accessed field. 😑

All your imported Logins will be inside the “Root” group. I’d suggest creating a new group (e.g. “Firefox Import”) and moving all imported Logins into it.

You can now open your default database and use the Database -> Merge From Database ... menu item, select the firefox_logins.kdbx file to include it’s contents into the current database. You’ll see a new “Firefox Imports” group show up.

You can now move the single entries into the appropriate groups if you want (e.g. “KeePassXC-Browser Passwords” if you use the browser extension).

Scaffolding for fetching and parsing emails from IMAP with Python

A friend asked me if I could help him write a Python script for fetching and processing data from emails in his mailbox … Well, the thing with emails is that they’re a pain to work with (in any form). So, I tried to help him out with a little scaffolding (also available as a Gist).

AR create_with

I knew there was AR’s 

find_or_create_by

  method but I didn’t know it had a very useful companion 

create_with

 .

@something ||= SomeModel.create_with(some: "stuff").find_or_create_by(foo: "bar")

This will either find a record 

SomeModel.where(foo: "bar")

  or create it with

SomeModel.create(foo: "bar", some: "stuff")

 . Very useful.

Backup And Restore Your Android Phone With ADB (And rsync)

Based on my previous scripts and inspired by two blog posts that I stumbled upon I tackled the “backup all my apps, settings and data” problem for my Android devices again. The “new” solutions both use

rsync

  instead of

adb pull

  for file transfers. They both use ADB to start a rsync daemon on the device, forward its ports to localhost and run rsync against it from your host.

Simon’s solution assumes your phone has rsync already (e.g. because you run CyanogenMod) and can become root via

adb root

. It clones all files from the phone (minus

/dev

 ,

/sys

 ,

/proc

  etc.). He also configures udev to start the backup automatically when the phone is plugged in.

pts solves the setup without necessarily becoming root. He also has a way of providing a rsync binary to phones that don’t have any (e.g. when running OxygenOS). He also has a few tricks on how to debug the rsync daemon setup on the phone.

I’ve tried to combine both methods. My approach doesn’t require adb or rsync to be run as root. It’ll use the the system’s rsync when available or temporarily upload and use a backup one extracted from Cyanogen OS (for my OnePlus One). Android won’t allow you to 

chmod +x

a file uploaded to

/sdcard

, but in

/data/local/tmp

it works. ?

The scripts will currently only backup and restore all of your 

/sdcard

directory. Assuming you’re also using something like Titanium Backup you’ll be able to backup and restore all your apps, settings and data. To reduce the amount of data to copy it uses rsync filters to exclude caches and other files that you definitely don’t want synced (

.DS_Store

  files anyone?).

At the moment there’s one caveat: I had to disable restoring modification times (i.e. use

--no-times

 ) because of an obnoxious error (they will be backuped fine, only restoring is the problem): ?

mkstemp “…” (in root) failed: Operation not permitted (1)

Additionally if you’re on the paranoid side you can also build your own rsync for Android to use as the backup binary.

The code and a ton of documentation can be found on GitHub. Comments and suggestions are welcome. ?