Howto Restore ZFS Encryption Hierarchies

Backing up encrypted ZFS datasets you’ll see that ZFS breaks up the encryption hierarchy. The backed up datasets will look like they’ve all been encrypted separately. You can still use the (same) original key to unlock all the datasets, but you’ll have to unlock them separately. 😐

This howto should help you bring them back together when you have to restore from a backup.

Assuming we’ve created a new and encrypted pool to restore the previous backup to (I’ll call it new_rpool). We send our data from the backup pool to new_rpool.

sudo zfs send -w -v backup/laptop/rpool/ROOT@zrepl_20210131_223653_000 | sudo zfs receive -v new_rpool/ROOT
sudo zfs send -w -v backup/laptop/rpool/ROOT/ubuntu@zrepl_20210402_113057_000 | sudo zfs receive -v new_rpool/ROOT/ubuntu
[...]

Note that we’re using zfs send -w which sends the encrypted blocks “as is” from the backup pool to new_pool. This means that these datasets can only be decrypted with the key they were originally encrypted with.

Also note that you cannot restore an encrypted root/pool dataset with another encrypted one: i.e. we can’t restore the contents/snapshots of rpool to new_rpool (at least not without decrypting them first on the sender, sending them unencrypted and reencrypting them upon receive). Luckily for me that dataset is empty. 😎

Anyway … our new pool should look something like this now:

$ zfs list -o name,encryption,keystatus,keyformat,keylocation,encryptionroot -t filesystem,volume -r new_rpool
NAME                   ENCRYPTION   KEYSTATUS    KEYFORMAT   KEYLOCATION  ENCROOT
new_rpool              aes-256-gcm  available    passphrase  prompt       new_rpool
new_rpool/ROOT         aes-256-gcm  unavailable  raw         prompt       new_rpool/ROOT
new_rpool/ROOT/ubuntu  aes-256-gcm  unavailable  raw         prompt       new_rpool/ROOT/ubuntu
[...]

Note that each dataset is treated as it is encrypted by itself (visible in the encryptionroot property). To restore our ability to unlock all datasets with a single key we’ll have to to some work.

First we have to unlock each of these datasets. We can do this with the zfs load-key command (my data was encrypted using a raw key in a file, hence the -L file:///...):

sudo zfs load-key -L file:///tmp/backup.key new_rpool/ROOT
sudo zfs load-key -L file:///tmp/backup.key new_rpool/ROOT/ubuntu
[...]

Although zfs load-key is supposed to have a -r option that works when keylocation=prompt it fails for me with the following error message 🤨:

sudo zfs load-key -r -L file:///tmp/backup.key new_rpool/ROOT

alternate keylocation may only be 'prompt' with -r or -a
usage:
        load-key [-rn] [-L <keylocation>] <-a | filesystem|volume>



For the property list, run: zfs set|get



For the delegated permission list, run: zfs allow|unallow

The keystatus should have changed to available now:

$ zfs list -o name,encryption,keystatus,keyformat,keylocation,encryptionroot -t filesystem,volume -r new_rpool
NAME                   ENCRYPTION   KEYSTATUS    KEYFORMAT   KEYLOCATION  ENCROOT
new_rpool              aes-256-gcm  available    passphrase  prompt       new_rpool
new_rpool/ROOT         aes-256-gcm  available    raw         prompt       new_rpool/ROOT
new_rpool/ROOT/ubuntu  aes-256-gcm  available    raw         prompt       new_rpool/ROOT/ubuntu
[...]

We can now change the encryption keys and hierarchy by inheriting them (similar to regular dataset properties):

sudo zfs change-key -l -i new_rpool/ROOT
sudo zfs change-key -l -i new_rpool/ROOT/ubuntu
[...]

When we list our encryption properties now we can see that all the datasets have the same encryptionroot. This means that unlocking it unlocks all the other datasets as well. 🎉

$ zfs list -o name,encryption,keystatus,keyformat,keylocation,encryptionroot -t filesystem,volume -r new_rpool
NAME                   ENCRYPTION   KEYSTATUS    KEYFORMAT   KEYLOCATION  ENCROOT
new_rpool              aes-256-gcm  available    passphrase  prompt       new_rpool
new_rpool/ROOT         aes-256-gcm  available    passphrase  none         new_rpool
new_rpool/ROOT/ubuntu  aes-256-gcm  available    passphrase  none         new_rpool
[...]

Restoring Dataset Properties

This howto doesn’t touch restoring dataset properties, because I’ve not been able to reliably backup dataset properties using the -p and -b options of zfs send. Therefore I make sure that I have a (manual) backup of the dataset properties with something like `zfs get all -s local > zfs_all_local_properties_$(date -Iminutes).txt`

Using WireGuard to Connect my IPv4 Network to the IPv6 Internet

For a long time I’ve wanted to have a proper IPv6 network at home. Although not always straight forward (because disabled by default or having less capable UI for configuration) I also wanted to be connected to the Internet via IPv6. Although all pieces of my network are IPv6-capable I’m stuck with an ISP that won’t enable it for DSL customers (not even as an “yes, I really want this and I’m willing to deal with the fallout” opt-in). I’ve skimmed NAT64 tutorials for OpenWRT over the years, but most of them were recommending software not updated since 2011. 😨 … and Internet hearsay says: NAT64 is lame! … literally! Implementations seem to prefer IPv4 in the presence of NAT64, because NAT64 is additional complexity on top of IPv4, hence assuming pure IPv4 will be faster).

So a proper IPv6 tunnel it is! It seems all but one 4to6 tunnel brokers have given up … and besides why give someone else your data? I figured a small (~3€/month) server and a little elbow grease would do the job as well. 🤠

The Home Situation

My home network consists of 3 UniFi Access Points, 2 UniFi Switches and a UniFi 3-Port Gateway. As a testament to Ubiquiti’s–at best mediocre–controller UI and basically non-scriptable configuration I’ve always hesitated to embrace it fully. I’ve kept my old–but still supported–OpenWRT router as my DNS server, because in the Unifi world DNS is not configurable beyond a “create DNS entries for DHCP hostnames” switch. Anyway … the Ubiquiti Gateway is connected to the DSL modem, so all IPv4 traffic goes that way.

Conventions

I’ll pretend my current IPv4 network is 10.11.12.0/24 and use the fd00:0011:0012::/48 ULA prefix for the IPv6 configuration. You should generate a random global ID and use it in your configuration. This is meant to prevent clashes when different (i.e. separately maintained) networks are merged at some point. The subnet ID (i.e. the 4th segment) still gives us enough leeway to segment our network and still have full /64s for each of them 😎. We’ll use the 13 segment for the home network and 14 for the WireGuard VPN.

By convention I’ll use .1 or ::1 to signify the UniFi gateway and .2 or ::2 to signify the OpenWRT router.

Configuration: UniFi Gear

The Ubiquiti gear needs to at least be aware of the networks that are going to be routed through it. This also means that we have to tell it we want to have an IPv6 network also. But in order to have more control over what comes next I made the OpenWRT router the DHCP server of the network.

UnifI Controller: network configuration

Notice that I turned off UniFi’s DHCP server and not activated any of the IPv6 DHCP or RA features here. A word of warning: whatever you put in the Gateway/Subnet fields UniFi will configure it’s gateway to have that IP address! … why would anyone ever use anything else as a gateway. 😒 Also we won’t need port forwarding, because we’ll have the OpenWRT router connect out.

Configuration: Cloud Server

I chose Hetzner Cloud for my VPN endpoint, as their servers have native IPv6 connections and every server gets a /64. Their smallest server costs around ~3€/month. I chose a Ubuntu 20.04 image and configured an SSH key. We then can log in and install WireGuard and something like UFW:

Now we can configure the WireGuard VPN endpoint. Note that we’ll only configure it for IPv6 connectivity!

Create a file /etc/wireguard/wg0.conf with the following content:

The ListenPort is arbitrary. We’ll use it across all peers in this WireGuard VPN for consistency. The PreUp lines make sure the kernel allows forwarding network packages in general. The PostUp lines allow packets from our VPN to reach the Internet as well as allowing the WireGuard connection through the firewall. %i will be replaced with the name of the WireGuard interface (wg0 in our case). You should replace eth0 with the name of the network interface of the host that is connected to the Internet.

Now we can start the WireGuard VPN endpoint and make sure it’s started on boot with the following commands. You can comment out the [Peer] section and start it now or wait until we’re done configuring the OpenWRT router also.

Configuration: OpenWRT Router

Connecting to the IPv6 Internet

First we have to install the necessary packages by following the WireGuard Basics guide on the OpenWRT Wiki.

On the “Network > Interfaces” page we add a new interface which I named “wan_wg” using the “WireGuard VPN” protocol. Generate a new private key with the wg genkey command and paste it into the “Private Key” field. Use the same “Listen Port” as for the Cloud VPN endpoint and add fd00:11:12:14::2/64 to the “IP Addresses“.

OpenWRT Router – WireGuard Interface: General Settings

In the “Advanced Settings” tab make sure “Use builtin IPv6-management” is activated.

In the “Firewall Settings” tab assign this interface to the “wan” zone.

In the “Peers” tab add a new peer and fill in the “Public Key” and “Preshared Key” fields with the contents of the wg.pub and wg.psk files of the Cloud VPN endpoint. The “Allowed IPs” should read ::/0. This may be the single most important configuration. This tells the router that if it doesn’t know where to send a specific IPv6 packet (i.e. because it’s not meant for a device on our home network) it should be sent here. Don’t add anything else otherwise the “Route Allowed IPs” option won’t add a route for ::/0 (ask me how I know 😑)! The “Endpoint Host” should contain a stable public IPv4 address or a hostname to reach the Cloud VPN server. In this case Hetzner Cloud servers are assigned a stable IPv4 address on creation. You can also use a dynamic DNS provider in case you don’t have a stable IP.

OpenWRT Router – WireGuard Interface: Peers

If we have done all the above steps and everything went fine … at this point we should be able to connect from the OpenWRT router out to the Internet via IPv6. We can try this out on the “Network > Diagnostics” page or on the command line.

We can successively test connecting to hosts and interfaces (via ping6 or traceroute6) “further” away:

  • ping6 fd00:11:12:14::2 # the OpenWRT router’s WireGuard address; this side of the tunnel is up
  • ping6 fd:11:12:14::3 # the Cloud VPN endpoint’s WireGuard address; we can reach the other end of the tunnel
  • ping6 <Cloud VPN endpoint's public IPv6 address> # we can get “out” of the tunnel
  • ping6 2a00:1450:4001:830::200e # we can connect to the IPv6 Internet (e.g. ipv6.google.com)
  • ping6 ipv6.google.com # even IPv6 name resolution works

Making our home network IPv6

Now that we can connect out (at least from the OpenWRT router) we still need to make the rest of our home network IPv6 aware. This is the reason I didn’t want the UniFi gear to be the DHCP server in the network any more. With OpenWRT we have much more control over what happens.

To achieve this we have to first tell OpenWRT our “IPv6 ULA-Prefix“. We do this on the “Network > Interfaces” page on the “Global network options” tab. As described above we use fd00:11:12::/48 here. Make sure the first 3 segments are the same on all the IPv6 addresses you configure with this tutorial!

After this we go back to the “Interfaces” tab and edit the “LAN” network. On the “General Settings” tab we set the “Protocol” to “Static address“, the “IPv6 assignment length” to 64, the “IPv6 assignment hint” to 13 (this will become the 4 segment in our IPv6 addresses) and “IPv6 suffix” to ::2.

OpenWRT – LAN Interface: General Settings

On the “Advanced Settings” tab we make sure to have “Use builtin IPv6-management” enabled.

Now we go to the “DHCP Server” tab. On its “General Setup” tab we make sure “Ignore interface” is disabled. On the “IPv6 Settings” tab set “Router Advertisement-Service” and “DHCPv6-Service” to “server mode“, “NDP-Proxy” to “disabled” and “DHCPv6-Mode” to “stateless + stateful“. The most important setting enabling “Always announce default router“. This makes sure that the OpenWRT router announces to the home network that it is the default router (for IPv6). This is necessary because we use local addresses (i.e. starting with fd...) instead of handing down the public prefix our Cloud VPN endpoint was assigned (this is a point I’m still working on 🚧).

OpenWRT – LAN Interface: DHCP IPv6 Settings

This should do the trick. Now other devices on the network should be getting IPv6 addresses from the fd00:11:12:13::/64 address range (check e.g. with ip -6 a). You may need to disconnect and then connect to the network again for the changes to be picked up.

Now you can do the same pinging/tracerouting procedure we did on the OpenWRT router.

More NAT

During my testing I was not able to ping the Cloud VPN endpoint’s WireGuard interface from my laptop. No amount of config jiggling on the OpenWRT router was helping getting the packages further than its WireGuard interface (i.e. fd00:11:12:14::2). Although I thought we could get by without using NAT … at least on the OpenWRT router (yes, I know we used masquerading on the Cloud VPN endpoint already 😓).

So I (begrudgingly) did what the NAT6 and IPv6 masquerading page on the OpenWRT Wiki said and lo and behold it worked. 😂

Shortcomings

I’m mostly unhappy about the (double) NAT. I’d like to get rid of it completely hopefully when I find a good way to hand down the public prefix assigned to the Cloud VPN endpoint. I’m still figuring out how to properly propagate through WireGuard and the home network (preferably) using the built-in prefix delegation mechanism. Help or ideas would be appreciated on this topic. 😅

My second point of grief is that all external traffic is routed through the <public prefix>::2 address although we have a whole /64 subnet. Somehow routing through the WireGuard network maps all address to ::2 negating the benefit of the masq6_privacy option on the “wan” firewall interface. 😞

Troubleshooting

Here are some tips to troubleshoot issues with this setup.

EUI64 addresses

You should put the last two segments of the LAN interface’s MAC address (looking like aa:bb:cc:dd:ee:ff) to memory as IPv6 devices use addresses derived from the MAC address (e.g something like fd00:11:12:13:a8bb:ccff:fedd:eeff or fe80::a8bb:ccff:fedd:eeff). Even if we’ve assigned specific addresses to specific devices or interfaces these addresses will still come up in routing or debugging output. There’re tools that help with converting MAC address + IPv6 network prefix into the final IPv6 addresses.

Checking IPv6 addresses

We can use ip -6 a to check what IPv6 address are assigned to interfaces on specific devices. Beware these can be quite a few. Check if the network prefixes (i.e. the first 4 segments of the IPv6 address) match what we have configured for our networks (+ the fe80:... addresses). Then we look at the host part (i.e. the last 4 segments of the IPv6 address) for the ones we’ve assigned manually or for the EUI64 addresses mentioned above. These will help us identify if our devices got any the IPv6 configuration right.

Checking IPv6 routes

We can use ip -6 r to check if our devices have the right routes configured. The most important one to look out for is the one starting with default via ... . It should point to one if the addresses of the OpenWRT router. We can also use traceroute6 to see what devices the packages go through.

More advanced debugging

Sometimes it helps watching what traffic goes through a specific interface. There’s an awesome blog post describing how to live capture packages from remote devices (using only tcpdump and ssh) and analyzing them in Wireshark. Our interface names would be wan_wg on the OpenWRT router or wg0 on the Cloud VPN endpoint.

Moving Passwords From Firefox (Lockwise) to KeePassXC

Go to about:logins in Firefox.

Click on the three dots in the top right corner, select “Export Logins…” and save your passwords info a CSV file (e.g. exported_firefox_logins.csv).

There’s one complication: Firefox will save dates as Unix timestamps (with millisecond resolution) which KeePassXC doesn’t understand, so we’ll have to help it out.

Save the following script into a file (e.g. firefox_csv_date_fix.py).

#!/usr/bin/env python3
#
# Author: Riyad Preukschas <riyad@informatik.uni-bremen.de>
# License: Mozilla Public License 2.0
# SPDX-License-Identifier: MPL-2.0


import csv
import sys

from datetime import datetime


def main():
    if len(sys.argv) != 2:
        print("Usage: {} <exported_firefox_logins.csv>".format(sys.argv[0]), file=sys.stderr)
        exit(1)

    csv_file_name = sys.argv[1]

    with open(csv_file_name, 'rt') as f:
        # field names will be determined from first row
        csv_reader = csv.DictReader(f)
        # read all rows in one go
        rows = list(csv_reader)

    # add new columns with Iso-formatted dates
    for row in rows:
        row['timeCreatedIso'] = datetime.fromtimestamp(int(row['timeCreated'])/1000).isoformat()
        row['timeLastUsedIso'] = datetime.fromtimestamp(int(row['timeLastUsed'])/1000).isoformat()
        row['timePasswordChangedIso'] = datetime.fromtimestamp(int(row['timePasswordChanged'])/1000).isoformat()

    # write out the updated rows
    csv_writer = csv.DictWriter(sys.stdout, fieldnames=rows[0].keys())
    csv_writer.writeheader()
    csv_writer.writerows(rows)


if __name__ == '__main__':
    main()

Call it giving it the path of the exported_firefox_logins.csv file and redirect the output into a new file:

python3 ./firefox_csv_date_fix.py ./exported_firefox_logins.csv > ./fixed_firefox_logins.csv

It will add new columns (named: old column name + “Iso”) with the dates converted to the ISO 8601 format KeePassXC wants.

We can now use the fixed_firefox_logins.csv file to import the logins into KeePassXC.

Select Database -> Import -> CSV File... from the menu.

It will ask you to create a new database. Just do it you can get the data into your default database later (e.g. call it firefox_logins.kdbx for now).

In the import form select “First line has field names” and match up KeePassXC’s fields with the columns from the CSV:

KeePassXC FieldFirefox CSV ColumnNotes
GroupNot Present
TitleurlFirefox doesn’t have a title field so we use the URL for it.
Usernameusername
Passwordpassword
URLurl
NoteshttpRealmI’m not sure how KeePassXC deals with HTTP authentication. There’s no special field for it, so I decided to keep it in the Notes field.
TOTPNot Present
IconNot Present
Last ModifiedtimePasswordChangedIsothe “Iso” part is important!
CreatedtimeCreatedIsothe “Iso” part is important!

Have a look at the preview at the bottom. Does the data look sane? 🤨

Sadly KeePassXC doesn’t let us import Firefox’s timeLastUsedIso field into its Accessed field. 😑

All your imported Logins will be inside the “Root” group. I’d suggest creating a new group (e.g. “Firefox Import”) and moving all imported Logins into it.

You can now open your default database and use the Database -> Merge From Database ... menu item, select the firefox_logins.kdbx file to include it’s contents into the current database. You’ll see a new “Firefox Imports” group show up.

You can now move the single entries into the appropriate groups if you want (e.g. “KeePassXC-Browser Passwords” if you use the browser extension).

“Grown-up” Chinese Jedi Elsa

Holy moly … CellSpex has the best, most eloquent (scathing) review of Disney’s live-action Mulan. Period. It starts with a hilariously satirical summary and goes on to dissect the maelstrom of bad decisions the film displays with its story adaptation (both from the folk tale as well as the Disney animated movie), characters, themes, style and production. I agree with each and every point.

Aktivieren Sie JavaScript um das Video zu sehen.
https://www.youtube.com/watch?v=ONVyLTBPz8E

iFlame 6S

I was mildly asleep as around 6am in the morning “someone” turned on a very bright light. There was a weird hissing sound. I woke up quite annoyed after around 10 seconds. But what I saw was beyond belief.

There was a roughly 30cm flame burning on my wife’s nightstand. I immediately went over and saw her iPhone on fire. I tried slapping it a few times which made the flames smaller so that I could disconnect the charging cable. I tried picking up the phone, but the case around was melting which burned my fingers. I then knocked it onto the ground (maybe not the smartest move), but I wanted to get the flames away from the lamp shade, plants and wall paper.

It seems the combination of disconnecting the charger and hitting the floor temporarily extinguished the flames. I immediately grabbed the phone at a point that seemed like it was not melted, ran to the kitchen and threw it into the sink. I let the water run over the phone for around a minute. I then filled a bowl with water and put the phone into it to make sure it doesn’t flare up again.

ٱلْحَمْدُ لِلَّٰهِ رَبِّ ٱلْعَالَمِينَ we got off lightly 😅 … the whole situation could have turned into a huge disaster. ☠

Tony Abbott and Tap Water

What could go wrong if-as a joke-your being asked if you could hack the former prime minister of Australia Tony Abbott? Well Alex Hope has documented it. Finding pictures of boarding pass he could log into the booking system of the airline (without additional authentication). Then he found out that the systems leaked sensitive information (passport number, telephone number, airline-internal comments about the passenger). He then went through the whole charade of finding someone in government responsible for concrete data security issues. 😵

There’s even an interesting section on when he finally gets through to Tony Abbot and they talk on a very personal level. Given the reason they were talking in the first place it also revolved about how complicated technology seems to be and how you learn how it works.

This lead Alex to reflect about how he started learning things and how you have to change your thinking when you are “hacking.” He gives a great example which he summarizes with:

In conclusion, to be a hacker u ask for tap water.

😂