Tag: Insights

Routing My Way Out With IPv6: NAT6

This article is part of a series of how I built a WireGuard tunnel for getting IPv6 connectivity. Where the last step was to figure out how to route packets from devices in my private network through the WireGuard tunnel to the Internet.

I’ve explored three different methods for solving this:

I’ll try to show how to set each of them up and try to convey their pros and cons.

TL;DR

You should always consider IPv6-PD first!

Consider any other option only if:

  • you have a “weird” setup or want to support an esoteric use case (like I do e.g. with too many local subnets for too long a public prefix)
  • you’re willing to set up, debug and maintain a somewhat experimental configuration
  • you more or less understand the tradeoffs
  • all of the above!

Starting Point

I’ll assume the following has been set up:

  • default OpenWRT networks named “LAN”, “WAN”, “WAN6”
  • default OpenWRT firewall rules
  • an IPv6 WireGuard tunnel with the endpoint on our OpenWRT router being 2000:30:40:50::2
  • the remote WireGuard tunnel end point forwards the whole 2000:30:40:50::/64 to our OpenWRT router

NAT6 a.k.a. Masquerading

NAT6 is basically a rehash of the “the old way” of using NAT for the IPv4 Internet. The router/gateway replaces the internal source (i.e. sender) address of a packet going out with its own public address. The router makes note of original sender and recipient to be able to reverse the process when an answer comes back. When the router receives a packet it forwards it to the actual recipient by replacing the destination address with the internal address of original sender.

Setup

On the “Network > Interfaces” page edit the “WAN6” interface and set “Protocol” to “unmanaged”. Then follow the OpenWRT NAT6 and IPv6 Masquerading documentation.

In my tests the masq6_privacy setting had no impact. All outgoing packages always had an address of 2000:30:40:50::2 (i.e. the router’s WireGuard interface address). 😕 It seems using WireGuard interferes with OpenWRT’s ability to generate temporary addresses for the interface. No amount of fiddling (e.g. setting addresses with /64, suffixes to “random”, setting prefix filters, setting a delegatable prefix, but disabling delegation, … I really got desperate) on the “WAN6_WG” interfaces’ settings or creating a “WAN6” alias and doing the same to it made the temporary addresses work. 😵 You could manually add addresses with random suffixes to the WireGuard interface … maybe even write a script that changes them periodically … 😅😐😞

Pros

  • multiple internal networks can be multiplexed onto one upstream network (even when the upstream prefix is too long (e.g. for IPv6-PD))
  • internal devices are not directly reachable from the Internet (this is not a replacement for a firewall!)

Cons

  • connections can only be started from internal devices
  • router needs to keep state for every connection
  • router needs to touch/manipulate every packet
  • you only have one static external address, because it seems temporary addresses (i.e. IPv6 privacy extensions) don’t work with WireGuard connections

Routing My Way Out With IPv6: IPv6-PD

Since I wrote my blog post about using a WireGuard tunnel for getting IPv6 connectivity there was one thing that was bugging me immensely: having to use NAT for IPv6. 😓

My initial howto used a private network for the WireGuard VPN which led to having two translation steps: one when entering the WireGuard VPN and one when exiting. I later realized I could use the global /64 assigned to the cloud VPN endpoint for the WireGuard VPN itself and just forward all traffic to and from it on the cloud VPN endpoint. This was easy, because the address mapping was 1:1 (cloud server’s /64 ⇔ WireGuard VPNs /64). This eliminated one translation.

The second translation (i.e. the one on the OpenWRT router) is more difficult to remove. The crux of the matter is that I only have a /64 for the tunnel which means I either have to select which internal network gets to be connected or I have to “multiplex” multiple internal /64s onto one upstream /64.

I’ve explored three different methods for solving this:

I’ll try to show how to set each of them up and try to convey their pros and cons.

TL;DR

You should always consider IPv6-PD first!

Consider any other option only if:

  • you have a “weird” setup or want to support an esoteric use case (like I do e.g. with too many local subnets for too long a public prefix)
  • you’re willing to set up, debug and maintain a somewhat experimental configuration
  • you more or less understand the tradeoffs
  • all of the above!

Starting Point

I’ll assume the following has been set up:

  • default OpenWRT networks named “LAN”, “WAN”, “WAN6”
  • default OpenWRT firewall rules
  • an IPv6 WireGuard tunnel with the endpoint on our OpenWRT router being 2000:30:40:50::2
  • the remote WireGurad tunnel end point forwards the whole 2000:30:40:50::/64 to our OpenWRT router

IPv6-PD (Prefix Delegation)

IPv6-PD (i.e. prefix delegation) is basically the built-in mechanism of sharing global IPv6 addresses with internal networks. As the word “delegation” implies you give away (a portion/sub-prefixes) to “downstream” networks. This also implies that if you get a long global prefix you may not be able to partition it for delegating it to (all) your internal networks. Normally you’ll get something like a /56 from your ISP, but I only have a /64, because I “hijacked” a cloud server’s addresses.

Setup

On the “Network > Interfaces” page edit the “WAN6” interface:

OpenWRT – WAN6 Interface: General Settings (for IPv6-PD)
OpenWRT – WAN6 Interface: General Settings (for IPv6-PD)
  • Set “Protocol” to “static”.
  • Set “Device” to “Alias interface: @wan6_wg”.
  • Set “IPv6 routed prefix” to the WireGuard public prefix (i.e. 2000:30:40:50::/64 in our case).
  • Make sure that in the “Advanced Settings” tab “Delegate IPv6 prefixes” is enabled.

After saving and applying those settings the “Network > Interfaces” page should look like the following screenshot.

OpenWRT – Resulting WAN6 Interfaces in Overview (for IPv6-PD)

Make sure that your WireGuard interface has its address set to 2000:30:40:50::2/128. If you have something like 2000:30:40:50::2/64 (note the /64) set as described in an earlier version of the previous how-to you’ll get the same /64 route for both the “WAN6” and the “WAN6_WG” interfaces. In my case packets from the “LAN” network would reach the Internet correctly, but the responses would arrive at the OpenWRT router’s WireGuard interface but never turn up in the “LAN” network. The following screenshot shows a working configuration on the “Status > Routes” page.

OpenWRT – Active IPv6 Routes (for IPv6-PD)

Pros

  • simple, built-in
  • devices can “directly” connect to the Internet (“no NAT, no nothin”; see below)
  • middleware boxes don’t need to keep state (it’s all just routing)
  • few things can break

Cons

  • your global prefix needs a short enough to be useful (i.e. shorter than /64)
  • internal devices have a routable address reachable from the Internet (i.e. your firewall should deny incoming connections from the Internet by default)

My First Container-based Postgres Upgrade

Yesterday I did my first container-based PostgreSQL version upgrade. In my case the upgrade was from version 13 to 14. In hindsight I was quite naïve. 😅

I was always wondering why distros kept separate data directories for different versions … now I know: you can’t do in-place upgrades with PostgreSQL. You need to have separate data directories as well as both version’s binaries. 😵 Distros have their mechanisms for it, but in the container world you’re kind of on your own.

Well not really … it’s just different. I found there’s a project that specializes in exactly the tooling part of the upgrade. After a little trial an error (see below) it went quite smoothly.

Procedure

In the end it came down to the following steps:

  1. Stop the old postgres container.
  2. Backup the old data directory (yay ZFS snapshots).
  3. Create the new postgres container (with a new data directory; in my case via Ansible)
  4. Stop the new postgres container.
  5. Run the upgrade. (see command below)
  6. Start the new postgres container.
  7. Run vacuumdb as suggested at the end of the upgrade. (see command below)

The Upgrade Command

I used the tianon/postgres-upgrade container for the upgrade. Since my directory layout didn’t follow the “default” structure I had to mount each version’s data directory separately.

docker run --rm \
-e POSTGRES_INITDB_ARGS="--no-locale --encoding=UTF8" \
-v /tmp/pg_upgrade:/var/lib/postgresql \
-v /tank/containers/postgres-13:/var/lib/postgresql/13/data \
-v /tank/containers/postgres-14:/var/lib/postgresql/14/data \
tianon/postgres-upgrade:13-to-14

I set the POSTGRES_INITDB_ARGS to what I used when creating the new Postgres container’s data directory. This shouldn’t be necessary because we let the new Postgres container initialize the data directory. (see below) I left it in just to be safe. 🤷

I explicitly mounted something to the container’s /var/lib/postgresql directory in order to have access to the upgrade logs which are mentioned in error messages. (see below)

The Vacuumdb Command

Upgrading finishes with a suggestion like:

Upgrade Complete
—————-
Optimizer statistics are not transferred by pg_upgrade.
Once you start the new server, consider running:
/usr/lib/postgresql/14/bin/vacuumdb –all –analyze-in-stages

We can run the command in the new Postgres container:

docker exec postgres vacuumdb -U postgres --all --analyze-in-stages

We use the postgres user, because we didn’t specify a POSTGRES_USER when creating the database container.

Pitfalls

When you’re not using the default directory structure there’re some pitfalls. Mounting the two versions’ data directories separately is easy enough … it says so in the README. It’s what it doesn’t say that makes it more difficult than necessary. 😞

Errors When Initializing the New Data Directory

The first error I encountered was that the new data directory would get initialized with the default initdb options. where I used an optimized cargo-culted incantation which was incompatible (in my case --no-locale --encoding=UTF8). The upgrade failed with the following error:

lc_collate values for database “postgres” do not match: old “C”, new “en_US.utf8”

So I made sure I created the new database container (with the correct initdb args) before the migration fixed this.

Extra Mounts for the Upgrade

What tripped me really up was that when something failed it said to look into a specific log file which I couldn’t find. 🤨 I had to also mount something to the /var/lib/postgres directory which then had all the upgrade log files. 😔

This also solved another of my problems where the upgrade tool wanted to start an instance of the Postgres database, but failed because it couldn’t find a specific socket … which also happens to be located in the directory mentioned above.

Authentication Errors After Upgrade

After the upgrade I had a lot of authentication errors although non of the passwords should have changed.

FATAL: password authentication failed for user “nextcloud”

After digging through the internet and comparing both the old and new data directories it looked like the password hashing method changed. It changed from md5 to scram-sha-256 (in pg_ hda.conf the line saying host all all all scram-sha-256). 😑Just re-setting (i.e. setting the same passwords again) via ALTER ROLE foo SET PASSWORD '...'; on all users fixed the issue.🤐

Using Posteo With Custom Domains

I tried to find out if you can use Posteo with a custom domain. And the short answer is: no.

Andy Schwartzmeyer found out the long answer. 👏😀 I implemented it for my blog and so far I’m happy. Interestingly enough it’s the receiving side that needs an external service. I currently use Forward Email‘s free tier, because I already have publicly known email addresses that I automatically collect emails from.

I always had the impression that sending mails is made more difficult in order to combat spammers, but what do I know. 🤷 Sending only needs to include Posteo in the SPF rules and adding a sender identity (usable after a one week cool down 🙈).

Skip unreadable files in awk scripts

Have you ever needed to give awk multiple files to process, but wanted it to skip files that it can’t open? Use this snippet!

BEGINFILE {
    # skip files that cannot be opened
    if ( ERRNO != "" ) {
        nextfile
    }
}

This trick is mentioned in the documentation for the BEGINFILE pattern, but not spelled out as code. You’re welcome. 😀

X: the dumbest OS you’ve ever seen

For future reference: Daniel Stone dispels myths about “great” X is and why it’s actually former X maintainers that designed and implemented Wayland.

Aktivieren Sie JavaScript um das Video zu sehen.
https://www.youtube.com/watch?v=GWQh_DmDLKQ

Multi-step Refactoring Pains in C++

Titus Winters talks about maintaining and refactoring large C++ code bases (i.e. code bodies that require multi-step refactoring). He describes how “higher-level” language features effectively make refactoring harder (e.g. functions, classes, templates, concepts).

Aktivieren Sie JavaScript um das Video zu sehen.
https://www.youtube.com/watch?v=v_yzLe-wnfk

Usefulness of Swap Explained

Chris Down explains how swap’s main role is being the missing backing store for anonymous (i.e. allocated by malloc) pages. While all other kinds of data (e.g. paged-in files) can be reclaimed easily and later reloaded, because their “source of truth” is elsewhere. There’s no such source for anonymous pages hence these pages can “never” be reclaimed unless there’s swap space available (even if those pages aren’t “hot”).

Linux has historically had poor swap (and by extension OOM) handling with few and imprecise means for configuration. Chris describes the behavior of a machine with and without swap in different scenarios of memory contention. He thinks that poor swap performance is caused by having a poor measure of “memory pressure.” He explains how work on cgroups v2 might give the kernel (and thus admins) better measures for memory pressure and knobs for dealing with it.