Recently my monitoring service warned me that my Raspberry Pi was not syncing its time any more. I logged into the devices and tried restarting systemd-timesyncd.service and it failed.
The error it presented was:
ConditionVirtualization=!container was not met
I was confused. Although I was running containers on this device, this was on the host! 😯
I checked the service definition and it indeed had this condition. Then I tried to look up the docs for the ContainerVirtualization setting and found out Systemd has a helper command that can be used to find out if it has been run inside a Container/VM/etc.
To my surprise running systemd-detect-virt determined it was being run inside a Podman container, although it was run on the host. I was totally confused. Does it detect any Container or being run in one? 😵💫
I tried to dig deeper, but the docs only tell you what known Container/VM solutions can be detected, but not what it uses to do so. So I searched the code of systemd-detect-virt for indications how it tried to detect Podman containers … and I found it: it looks for the existence of a file at /run/.containerenv. 😯
Looking whether this file existed on the host I found out: it did!!! 😵 How could this be? I checked another device running Podman and the file wasn’t there!?! 😵💫 … Then it dawned on me. I was running cAdvisor on the Raspberry Pi and it so happens that it wants /var/run to be mounted inside the container, /var/run just links to /run and independent of me mounting it read-only it creates the /run/.containerenv file!!! 🤯
I looked into /run/.containerenv and found out it was empty, so I removed it and could finally restart systemd-timesyncd.service. The /run/.containerenv file is recreated on every restart of the container, but at least I know what to look for. 😩