Incident Overview

On 23th October, 2026, the Kraftland core server experienced boot issues after a kernel upgrade. This blog covers the timeline and aftermath analysis about this incident.

Timeline

Due to recent security concerns, we decided to upgrade the Linux kernel.

At 2026-10-23 22:15, we updated the whole server OS via pacman -Syu and called reboot afterwards.

At 22:20, after waiting for several minutes and not getting an SSH shell, we noticed that something was wrong. The server has been accessed directly and there are multiple errors about disk mounting. Due to the complex nature of our disk setup, it was not clear at the time which cluster caused the failure.

At 22:30, we verified locally that all disk clusters are operating normally on another machine.

At 22:32, a live Arch Linux ISO secure boot compliant drive was set up and our server booted into recovery mode. All drives were then mounted manually in the live environment. We confirmed that the Unified Kernel Image is not faulty, and the version matches root filesystem kernel.

At 22:35, we determined that the most probable failure point is the root filesystem after analysing mount units and fstab. As such, a decision was made to make use of UAPI.2 The Discoverable Partitions Specification. The root filesystem on Kraftland servers are RAID arrays for redundancy, as Btrfs supports mounting a cluster by specifying any volume inside, we made the primary drive as boot drive by setting the correct partition type.

At 22:37, the system was rebooted. It was found to boot with read-only root filesystem, despite the specification of rw on kernel command line in /etc/kernel/cmdline.

At 22:40, we discovered that the EFI System Partition was not mounted properly, and /proc/cmdline returns @/etc/kernel/cmdline rather than standard moeOS kernel command line.

At 22:48, the server was configured correctly to mount EFI System Partition manually. The kernel command line was rectified too. Thus all Kraftland services went back to normal operation again.

Aftermath

Having a 30-minute maintenance was practically unheard of. We are committed to reduce chances of interrupting downtime. The issue has been analysed closely, and we believe it is de-mystified. Should a kernel upgrade happen again, the underlying issue would not emerge.

Kernel command line mismatch

As mentioned above. While troubleshooting, we found out that kernel command line was set to @/etc/kernel/cmdline rather then the contents of said file. This may be very confusing at first, since the moeOS project has been using the exact same uki.conf configuration for systemd Ukify since January.

A little more digging in Ukify’s code shows that it is not unquotting those ini-formatted files. Leading the @/path/to/cmdline syntax to be interpreted as literal strings. But something doesn’t sound right. If it never did support unquoting of configuration entries, then why didn’t it break at the beginning?

The harsh truth is: another component that was calling Ukify, systemd kernel-install was silently ignoring the Cmdline= unconditionally since the beginning, as such this setup did not break in January - September: it was simply re-using current cmdline rather than honouring user settings. It was not until systemd Pull Request #43781 that kernel-install finally started using user-specified values and broke our setup where it was quoted.

Read-only root filesystem and other mount shenanigans

With cmdline issue debunked, it is fairly easy to find out that missing rw, rootflags parameter and fstab entry caused systemd to mount a read-only root. Should the kernel command line issue be rectified, this problem will not emerge again.

The ESP auto-mount issue was a plain oversight. Since our root was on multiple btrfs devices spanning multiple disks, systemd-gpt-auto-generator would refuse automatic dissection, as specified in the manual:

Or in other words: automatic discovery of the root file system and of the non-root file systems are independent operations, that do not rely on each other, and are done during two distinct phases of the boot process (one in the initrd, the other after). These partitions will not be searched for on systems where the root file system is distributed on multiple disks, for example via btrfs RAID.

and we can confirm it live by running it manually:

➜  ~ SYSTEMD_LOG_LEVEL=debug /usr/lib/systemd/system-generators/systemd-gpt-auto-generator /tmp/tmp.ulemvloHXk /tmp/tmp.ulemvloHXk /tmp/tmp.ulemvloHXk
Failed to read $container of PID 1, ignoring: Permission denied
Found container virtualization none.
Reading EFI variable /sys/firmware/efi/efivars/LoaderDevicePartUUID-4a67b082-0a4c-41cf-b6c7-440b29bb8c4f.
Adding /: /dev/gpt-auto-root fstype=(any)
Failed to create unit file '/tmp/tmp.ulemvloHXk/-.mount': File exists
Failed to open '/usr/.systemd-sysext': No such file or directory
Neither root nor /usr/ file system are on a (single) block device.
Skipping automatic GPT dissection logic, root file system not backed by a (single) whole block device.

Moral of the story

While upgrades failing due to dependency changes, transient network issues and many other factors are unfortunately inevitable, the ace in the hole was to recover from failures faster. There is now always a recovery boot disk around, and upgrades are watched more closely. In the end, we learned a lot about systemd-gpt-auto-generator, albeit not at a great point of time 👻.