Gripes With Setting Up My Unifi UDM

I’ve recently purchased an UDM SE when it was on sale. I use it as my main router moving away from OpenWRT (for this role). Especially the “recent” improvements including the zone-based Firewall configuration as well as beginnings of “usable” IPv6 support are what allowed me to make the jump.

I first chose to import my old self-hosted Unifi Network setup, but chose to redo it from scratch, because it seemed “buggy.”

I’m happy overall. The new zone-based firewall is a HUGE improvement (there’re specifics below)!

My gripes1 with the setup (as of January 2026):

You cannot–under any circumstance–create a working allow rule from a network in the Guest zone … there seem to be “hidden rules” that prevent this. Finding this out ate a whole weekend.

Firewall rules can’t have “any” zone as a source or destination. E.g. you can’t create a pure WAN egress rule.

Using the object policies you can get an error, saying you can’t create any more “ACL rules.” … why?!? I’m not using ACL rules (knowingly)! How many can I use (IIRC I had four or five)? How do I find out which ones they are? They count even when they are all paused?!?!?! ☠️

If you want to use the zone-based Firewall to allow Internet access to specific domains only make sure your UDM/etc. is the device’s DNS server. It doesn’t work with external DNS servers.

The Intrusion Prevention System blocks connections (e.g. www.privacy-handbuch.de) even when it’s set to only “notify.” In the logs it doesn’t say what the reason for blocking was, I just found out by elimination. 🤮

It seems not all blocked connections are shown in the flows/logs. I’ve had to create firewall rules for devices and services that were blocked, but didn’t show up in the flows/logs view (even with all the extended logging settings set). I only found out because of my internal monitoring setup (yay, Prometheus Blackbox Exporter and Ping Exporter). 😱

You cannot use device groups in Firewall rules, only in object policies.

You can select devices as sources in Firewall policies, but not as destinations.

You can’t add comments to Port or IP lists. Neither on the whole list, nor on the individual entries.

“Add multiple” fields won’t filter duplicates automatically … they will nag you until you’ve removed them manually. 😞

There’s no way to bulk export or import for DNS records … or firewall rules.

You can’t use IPv6 in WireGuard VPNs! 🤬

You can’t change the settings of the WireGuard Server or Clients. I know why they don’t allow it, but it’s rubbing me the wrong way.

MLO is a lie!

  1. according to the principle: “if you want to nag at least have the courtesy to be specific!” ↩︎

Kubernetes Resource Requests Are a Massive Footgun

If you have Kubernetes workloads that configure Resource Requests on Pods or Containers there’s a footgun “hidden” in a sentence in the documentation (kudos if you spot it immediately):

[…] The kubelet also reserves at least the request amount of that system resource specifically for that container to use. […]

This means Resource Requests actually reserve the requested amount of resources exclusively. To emphasize: this is not a fairness measure in case of over-provisioning! So, if there are Resource Requests you can’t “overprovision” your node/cluster … hell, the new pod won’t even be scheduled although your node is sitting idle. 😵😓

By the time you find out why and have patched the offending resources you’ll be swearing up and down. 🤬

Oh … and wait till you see what the Internat has to say about Resource Limits. 😰

Simulating Statically Compiled Binaries in Glorified Tarballs

Containers won for one reason: they simulate a statically compiled binary that’s ergonomic for engineers and transparent to the application. A Docker image is a glorified tarball with metadata in a JSON document.

From Joseph’s comment on “Containers and giving up on expecting good software installation practices”

I hadn’t thought of it that way, but from a developer’s perspective it makes sense. It may not be incidental that the new programming languages of the 2010s (e.g. Go, Rust, Zig) produce statically linked binaries by default.

I always thought of containers as a way to add standardized interfaces to an application/binary that can be configured in a common way (e.g. ports, data directories, configurationenv vars, grouping and isolation). The only other ecosystem that does this and maybe even goes a little further is Nix.

Because the binary format itself is ossified and the ecosystem fragmented enough we missed the train for advanced lifecycle hooks for applications (think multiple entry points for starting, pausing, resuming, stopping, reacting to events, etc. like on Android, iOS, MacOS) … in Linux this is something that’s again bolted on from the outside: with e.g. D-Bus, Systemd, CRIU).

Configuring Custom Ingress Ports With Cilium

This is just a note for anyone looking for a solution to this problem.

While it’s extremely easy with the Kubernetes’ newer Gateway API via listeners on Gateway resources it seems the Ingress resources were always meant to be used with (global?) default ports … mainly 80 and 443 for HTTP and HTTPS respectively. So every Ingress Controller seems to have their own “side-channel solution” that leverages some resource metadata to convey this information. For Cilium this happens to be the sparsely documented ingress.cilium.io/host-listener-port annotation.

So your Ingress definition should look something like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ...
  namespace: ...
  annotations:
    ingress.cilium.io/host-listener-port: 1234
spec:
  ingressClassName: cilium
  rules:
  - http: ...

Fixing Dracut for Encrypted ZFS on Root on Ubuntu 25.10

I just upgraded from Ubuntu 25.04 to 25.10 … well it was more of a reinstall really. Because I knew the new release changed the initrd-related tools to Dracut I tried to understand all the changes from a test installation in a VM. Well, I still somehow broke Dracut’s ability to unlock my encrypted ZFS on root setup automatically.

Looking at journalctl it claimed it couldn’t find the key file:

dracut-pre-mount[940]: Warning: ZFS: Key /run/keystore/rpool/system.key for rpool/enc hasn't appeared. Trying anyway.
[...]
dracut-pre-mount[1001]: Key load error: Failed to open key material file: No such file or directory
[...]
systemd[1]: Mounting sysroot.mount - /sysroot...
mount[1007]: zfs_mount_at() failed: encryption key not loaded
systemd[1]: sysroot.mount: Mount process exited, code=exited, status=2/INVALIDARGUMENT
systemd[1]: sysroot.mount: Failed with result 'exit-code'.
systemd[1]: Failed to mount sysroot.mount - /sysroot.
systemd[1]: Dependency failed for initrd-root-fs.target - Initrd Root File System.

All I could do was mounting the keystore manually in the emergency console:

systemd-cryptsetup attach keystore-rpool /dev/zvol/rpool/keystore
mkdir -p /run/keystore/rpool
mount /dev/mapper/keystore-rpool /run/keystore/rpool

After pressing Ctrl-d Systemd continued booting as if everything was OK. This worked, but was HUGELY annoying, especially considering it was also using an English keyboard mapping. 🤬

After I was done setting up my desktop I took the time investigate the issue. I compared all the things between my real system and the freshly setup VM. After comparing the system startup plots (exported with systemd-analyze plot > plot.svg) I noticed that the systemd-ask-password.service would start quite late in my real system (after I manually mounted the keystore). I knew there was a bug report for teaching Dracut Ubuntu’s ZFS on root encryption scheme (i.e. putting the root ZFS dataset’s encryption keys in a LUKS container on a Zvol (rpool/keystore)). So I looked at the actual patch and tried to walk through of how it would behave on my system. There I noticed that the script actually assumes the ZFS encryption root to be the same as the Zpool’s root dataset (e.g. rpool). 😯 I moved away from this kind of setup years ago as it makes restoring from a backup quite cumbersome. So I was using a sub-dataset for the encrypted data (e.g. root/crypt) which messed up the logic which assumed it to only contain the pool name. 🤦‍♂️

Long story short the following patch determines the pool name of the encryption root before trying to open and mount the LUKS keystore:

--- zfs-load-key.sh.orig        2025-10-16 20:44:47.955349974 +0200
+++ zfs-load-key.sh     2025-10-16 20:55:00.229000464 +0200
@@ -54,9 +54,11 @@
     [ "$(zfs get -Ho value keystatus "${ENCRYPTIONROOT}")" = "unavailable" ] || return 0

     KEYLOCATION="$(zfs get -Ho value keylocation "${ENCRYPTIONROOT}")"
+    # `ENCRYPTIONROOT` might not be the root dataset (e.g. `rpool/enc`)
+    ENCRYPTIONROOT_POOL="$(echo "${ENCRYPTIONROOT}" | cut -d/ -f1)"
     case "$KEYLOCATION" in
-        "file:///run/keystore/${ENCRYPTIONROOT}/"*)
-            _open_and_mount_luks_keystore "${ENCRYPTIONROOT}" "${KEYLOCATION#file://}"
+        "file:///run/keystore/${ENCRYPTIONROOT_POOL}/"*)
+            _open_and_mount_luks_keystore "${ENCRYPTIONROOT_POOL}" "${KEYLOCATION#file://}"
             ;;
     esac

🎉

Running k3s on Incus

I know the pain to manage a bunch of services on my own. Even with relying on Incus, Podman and Systemd as much as possible held together by lot’s of Ansible duct tape: it’s still arduous. I convinced myself change was in order: … something something Kubernetes.

My main criteria are basically:

  • Must be able to run on a single node (for now). i.e. no clustered services or databases. (k3s looks like it fits the bill)
  • Services must be able to be deployed with public service definitions (Helm FTW)
  • These service definitions must lend themselves to be version controlled
  • All relevant data directories must live on a separate ZFS datasets

Running k3s in an Incus container

You can run k3s in an Incus container, but it gets increasingly difficult. There’re reports of people getting it to run, but it gets increasingly difficult. Even public LXD/LXC definitions for microk8s or k3s are either quite old (as of 2025-08 3 and 6 years old respectively) and blast HUGE holes in the sandbox. ☹️ K3s “requires” access to /dev/kmsg, several places in /proc and /sys as well as modprobing several kernel modules (it checks for access to them and spams the logs with warnings and errors). 😶

It looks doable in a technical sense, but it’s a huge pain having to go though Incus, without any of the (sandboxing/security) benefits. So the general wisdom is to just use a VM. (No, I didn’t try k3s’ experimental rootless mode)

Running k3s in an Incus VM

I started with a fresh VM and could reuse my now much simplified Ansible tasks for setting um k3s. But my happiness got cut short by the k3s service spamming the journal with useless

level=error msg="failed to ping connection: disk I/O error: no such device"

error messages.After removing all the directories and files from /var/lib/rancher/k3s and starting the server by hand I got:

Error: preparing server: failed to bootstrap cluster data: creating storage endpoint: failed to create driver for default endpoint: setup db: disk I/O error: no such device

Some more mucking around with the k3s server config revealed a puzzling, but more useful

failed to mount overlay: invalid argument.

Looking at what dmesg had to say I got:

overlayfs: upper fs does not support tmpfile.
overlayfs: failed to set xattr on upper
overlayfs: …falling back to redirect_dir=nofollow.
overlayfs: …falling back to uuid=null.
overlayfs: …falling back to xino=off.
overlayfs: try mounting with 'userxattr' option
overlayfs: upper fs missing required features.

Long story short: it turns out in my eagerness I had mounted a custom Incus volume as k3s’ data directory (/var/lib/rancher/k3s). This being a VM (instead of a container) it mounted the volume using the virtiofs protocol. And it turns out the overlayfs doesn’t like being put on top of virtiofs devices (or NFS it seems). 😵‍💫 But good news: it was fixable, although hacky. I found out by grepping for “virtiofsd” processes that Incus vendors its own virtiofsd binary in /opt/incus/bin/virtiofsd. And it already runs it with the --posix-acl option with implies the required --xattr option. But Incus currently doesn’t support any way for configuring virtiofsd. 😓 So the only solution (by the main Incus maintainer none the less) is to replace /opt/incus/bin/virtiofsd with a shim script calling the real virtiofsd binary with the additional --modcaps=+sys_admin option. Basically something silly like:

#!/usr/bin/bash
exec /opt/incus/bin/virtiofsd.orig --modcaps=+sys_admin "$@"

Yeah also, “try mounting with ‘userxattr’ option” was not helpful and sent me down the wrong path. 🤐

All in all … all these stumbling blocks ate my weekend. Which was kind of in line with my prejudices against Kubernetes. 😅

Force VLC to use VA-API for Hardware Accellerated Video Decoding

tl;dr: add the --avcodec-hw=vaapi option on the command line or to the Exec option in the .desktop file.

It’s stupid, I know, but it’s been bothering me for a while now. Especially when I want to watch conference talks that are available in the AV1 video format (e.g. FOSDEM) the video always seems to hang (show an old frame indefinitely), have broken decoding (shows alternating weirdly colored blocks), de-sync from audio or just stay black. This is happening on both Intel and AMD integrated graphics for years now, and I somehow decided that VDPAU must be the culprit. I also definitely know that VA-API works on my machines, because I’ve tested it … so that can’t be the problem. 😇

VLC (generally) supports both VA-API (mainly for Intel and AMD hardware) and VDPAU (for Nvidia) libraries for hardware accelerated video decoding, but on my Ubuntu desktop machines prefers VDPAU on any hardware for some reason. The settings don’t even show support for anything else: “Simple Preferences” -> “Input/Codecs” tab -> “Hardware-accelerated decoding” only shows “Automatic”, “VDPAU video decoder” and “Disable” options. 😵‍💫 The only “variant” that correctly uses VA-API automatically on my machines is the VLC Flatpak. I checked which backend was used via the “Modules Tree” tab in the “Tools” -> “Messages” dialog. It will show “vdpau”-something in the “video output” subtree (or not).

The Solution

So I dug through weird forums and tried different suggested options, of those many weren’t even supported until I found the right incantation: --avcodec-hw=vaapi .

Fixing the .desktop file

To make your desktop always call VLC with the right options we have to edit VLC’s so-called .desktop file. Mine was located in /usr/share/applications/vlc.desktop.
The relevant line looked like this: Exec=/usr/bin/vlc --started-from-file %U .

Copy the vlc.desktop file to either the $HOME/.local/share/applications/ directory if you want to change the behavior only for you. Alternatively if you have root privileges you can update vlc.desktop for all users of that machine by copying it to /usr/local/share/applications/ . NOTE: you may need to create those directories first.

Then edit the Exec= line to look like this: Exec=/usr/bin/vlc --avcodec-hw=vaapi --started-from-file %U

Or if you want to just copy the relevant commands:

# create the directory for personal .desktop files
mkdir -p $HOME/.local/share/applications/

# copy the original vlc.desktop to this directory
cp /usr/share/applications/vlc.desktop $HOME/.local/share/applications/

# edit the copied vlc.desktop by changing its "Exec" option to include the relevant VLC option
desktop-file-edit --set-key=Exec --set-value="/usr/bin/vlc --avcodec-hw=vaapi --started-from-file %U" $HOME/.local/share/applications/vlc.desktop

Enjoy!

Running Circles Around Detecting Containers

Recently my monitoring service warned me that my Raspberry Pi was not syncing its time any more. I logged into the devices and tried restarting systemd-timesyncd.service and it failed.

The error it presented was:

ConditionVirtualization=!container was not met

I was confused. Although I was running containers on this device, this was on the host! 😯

I checked the service definition and it indeed had this condition. Then I tried to look up the docs for the ContainerVirtualization setting and found out Systemd has a helper command that can be used to find out if it has been run inside a Container/VM/etc.

To my surprise running systemd-detect-virt determined it was being run inside a Podman container, although it was run on the host. I was totally confused. Does it detect any Container or being run in one? 😵‍💫

I tried to dig deeper, but the docs only tell you what known Container/VM solutions can be detected, but not what it uses to do so. So I searched the code of systemd-detect-virt for indications how it tried to detect Podman containers … and I found it: it looks for the existence of a file at /run/.containerenv. 😯

Looking whether this file existed on the host I found out: it did!!! 😵 How could this be? I checked another device running Podman and the file wasn’t there!?! 😵‍💫 … Then it dawned on me. I was running cAdvisor on the Raspberry Pi and it so happens that it wants /var/run to be mounted inside the container, /var/run just links to /run and independent of me mounting it read-only it creates the /run/.containerenv file!!! 🤯

I looked into /run/.containerenv and found out it was empty, so I removed it and could finally restart systemd-timesyncd.service. The /run/.containerenv file is recreated on every restart of the container, but at least I know what to look for. 😩

Update 2025-03-23: I’ve created a bug report: https://github.com/containers/podman/issues/25655

Dropbear vs SSH woes between Ubuntu LTSes

Imagine you’re using dropbear-initrd to log in to a server during boot for unlocking the hard disk encryption and you’re greeted with the following error after a reboot:

root@server: Permission denied (publickey).

🤨😓😖 You start to sweat … this looks like extra work you didn’t need right now. You try to remember: were there any updates lately that could have messed up the initrd? … deep breath, lets take it slowly.

First try to get SSH to spit out more details:

$ ssh -vvv server-boot
[...]
debug1: Next authentication method: publickey
debug1: Offering public key: /home/user/.ssh/... RSA SHA256:... explicit
debug1: send_pubkey_test: no mutual signature algorithm
[...]

That doesn’t seem right … this worked before. The server is running Ubuntu 20.04 LTS and I’ve just upgraded my work machine to Ubuntu 22.04 LTS. I know that Dropbear doesn’t support ed25519 keys (at least not on the version on the server), that’s why I still use RSA keys for that. 🤔

Time to ask the Internet, but all the posts with a “no mutual signature algorithm” error message are years old … but most of them were circling around the SSH client having deprecated old key types (namely DSA keys). 😯

Can it be that RSA keys have also been deprecated? 😱 … I’ve recently upgraded my client machine 😶 … no way! … well, yes! That was exactly the problem.

Allowing RSA keys in the connection settings for that server allowed me to log in again 😎:

PubkeyAcceptedKeyTypes +ssh-rsa

But this whole detour unnecessarily wasted an hour of my life. 😓

Finding out what rules to add to /etc/gai.conf

I had a weird problem. I was using network prefix translation (NPT) for routing IPv6 packets to the Internet through a VPN. But while all devices could connect to the IPv6 Internet without problems, they never did so on their own. They always preferred IPv4 connections when they had the choice. 🤨

Problem Background

I knew that modern network stacks are configured to prefer IPv6 over IPv4 generally, but was baffled why it wouldn’t use IPv6 since it was clear that connections to the Internet work. A little bit of tinkering revealed that IPv4 connections to the Internet are preferred only when my device had no global IPv6 addresses. Because I was relying on NPT my devices only had ULAs.

It turns out that the wise people making standards decided that when a device has only private IPv4 addresses and ULAs IPv4 connections are preferred for the Internet under the assumption that private IPv4 addresses are definitely NATed while IPv6’s ULA probably (definitely?) won’t. 😯

Finding a Solution

A quick search for anything related to IPv4 vs. IPv6 priority leads exclusively to questions and posts where the authors want to always have IPv4 prioritized over IPv6. Although my case was the opposite one thing became clear: it had to do with modifying /etc/gai.conf. It’s a file for configuring RFC 6724 (i.e. Default Address Selection for Internet Protocol Version 6 (IPv6)).

This allowed me to influence the selection algorithm which seemed to be needed for solving my problem. If you open this file it even has commented-out lines for solving the “always prefer IPv4 over IPv6” problem. The inverse case was not so simple, because among the precedence rules there was no address range for ULAs and adding one for my specific ULA didn’t solve the problem either:

[...]
# precedence <mask> <value>
# Add another rule to the RFC 3484 precedence table. See section 2.1
# and 10.3 in RFC 3484. The default is:
#
precedence ::1/128 50
precedence ::/0 40
precedence 2002::/16 30
precedence ::/96 20
precedence ::ffff:0:0/96 10
precedence 3fff:01:23::/48 45 # <-- added my ULA, but didn't help
#
# For sites which prefer IPv4 connections change the last line to
#
#precedence ::ffff:0:0/96 100
[...]

Manual Algorithm

I tried to take a step back and find out if a precedence setting was even the right change. I bit the bullet and tried to evaluated the “Source Address Selection” algorithm from RFC 6725 (Section 5) by hand.

Candidate Addresses

My candidate addresses for the destination (this server) were:

2a01:4f8:c2c:8101::1   # native IPv6
::ffff:116.203.176.52  # native IPv4 (mapped to IPv6 for this algorithm)

My candidate source addresses (from my WLAN connection) looked like:

3fff:01:23::aa  # global dynamic noprefixroute
3fff:01:23::bb # global temporary dynamic
3fff:01:23::cc # global mngtmpaddr noprefixroute
::ffff:10.0.0.50 # private IPv4 (mapped to IPv6 for this algorithm)

The Rules

Rule 1: Prefer same address.

skip, source and destination are not the same.

Rule 2: Prefer appropriate scope.

skip, connection is unicast, so no multicast.

Rule 3: Avoid deprecated addresses.

skip, no deprecated source addresses used.

Rule 4: Prefer home addresses.

skip? I was not sure what a “home address” is supposed to be, but it seems related to mobile networks. I just assumed all source addresses were “home” addresses.

Rule 5: Prefer outgoing interface.

skip, I was already only considering the outgoing interface here.

Rule 5.5: Prefer addresses in a prefix advertised by the next-hop.

skip? all next-hops were fe00::<router's EUI64>.

Rule 6: Prefer matching label.

We get the default labels from /etc/gai.conf (mine from Ubuntu 21.10):

[…]
#label ::1/128       0  # loopback address
#label ::/0          1  # IPv6, unless matched by other rules
#label 2002::/16     2  # 6to4 tunnels
#label ::/96         3  # IPv4-compatible addresses (deprecated)
#label ::ffff:0:0/96 4  # IPv4-mapped addresses
#label fec0::/10     5  # site-local addresses (deprecated)
#label fc00::/7      6  # ULAs
#label 2001:0::/32   7  # Teredo tunnels
[…]

Then the destination addresses would get labeled like this:

2a01:4f8:c2c:8101::1   # label 1
::ffff:116.203.176.52  # label 4

And the source addresses would get labeled like this:

3fff:01:23::aa  # label 6
3fff:01:23::bb # label 6
3fff:01:23::cc # label 6
::ffff:10.0.0.50 # label 4

Here we see why IPv4 addresses are selected: their destination and source addresses have the same label while the IPv6 address don’t. 😔

So I could add a new label for our ULA that has the same label as the ::/0 addresses (i.e. 1 here). I didn’t change the label on the fc00::/7 line in order not to change the behavior for all ULAs, but I wanted a special rule for my specific network. So I uncommented the default label lines and added the following line:

label 3fff:01:23::/48 1  # my ULA prefix and the same label as ::/0

Reboot (may no be strictly necessary) … and lo and behold it worked! 😎

Conclusion

While this worked I really felt uneasy messing with the address priorization especially if you take into account that I’d have to do this on every device. This is on top of the already esoteric setup for using NPT. 🙈

I later found out that when the VPN goes down (i.e. there’s no IPv6 Internet connectivity) it won’t (actually can’t) fall back to IPv4 for the Internet connection. 😓

Update 2025-12-14: Use 3fff::/20 as documentation prefix.