raspberry-pi-mixer/docs/research/base-os-decision.md

Base OS Selection — Buildroot/Yocto/RPiOS Lite + RT Kernel

Date: 2026-05-19 Status: Approved Parent: P1-R1 (System Research Report) Next: P2 (Kernel Build)

---

Executive Summary

Decision: Raspberry Pi OS Lite (64-bit, Bookworm) + custom PREEMPT_RT kernel.

RPiOS Lite is selected over Buildroot and Yocto for this project. It provides the Debian/bookworm ecosystem for rapid audio-stack iteration, active Foundation maintenance, and straightforward custom kernel integration. Buildroot and Yocto are evaluated below and rejected for reasons of development velocity (Buildroot requires full-image rebuild per package change) and disproportionate complexity (Yocto's ~50GB build footprint and steep learning curve are unjustified for a single-purpose audio appliance with one developer).

---

1. Build System Comparison

1.1 Buildroot

Aspect	Assessment
Version	2024.02+ (rolling releases every 3 months)
RPi4 64-bit support	First-class: raspberrypi4_64_defconfig
Build time (clean)	~30-45 min on modern x86 (8-core)
Build output	~120MB EXT4 rootfs image + FAT32 boot partition
Init system	BusyBox init (default) or systemd (optional)
Cross-compilation	Bootlin external toolchain (aarch64 glibc stable)
Package format	None at runtime — every change requires image rebuild
Kernel config	BR2_LINUX_KERNEL_CUSTOM_TARBALL → RPi kernel tarball
PREEMPT_RT support	Via kernel config fragment overlay (BR2_LINUX_KERNEL_CONFIG_FRAGMENT_FILES)
Read-only rootfs	SquashFS built-in (BR2_TARGET_ROOTFS_SQUASHFS), initramfs/OverlayFS support

Verdict for this project: Rejected.

Buildroot excels at producing minimal, reproducible appliance images. However, the "rebuild image to change a package" model is a poor fit for audio development where you need to iterate on JACK configs, test different DSP toolchains, and install debugging tools rapidly. The 30-min rebuild cycle per package tweak would severely hamper velocity. Buildroot is best for the "final appliance image" stage — it could be revisited for production deployment after development stabilizes on RPiOS Lite.

1.2 Yocto Project / OpenEmbedded

Aspect	Assessment
Version	Scarthgap 5.0 (LTS, April 2024) or Styhead 5.1
RPi4 64-bit support	First-class via meta-raspberrypi layer (maintained by Andrei Gherzan)
Build time (clean)	~2-4 hours first build; ~15-30 min incremental
Build footprint	~50 GB disk for build directory
Init system	systemd (default in poky)
Cross-compilation	Built-in toolchain generation (cross + native + nativesdk)
Package format	opkg, rpm, or deb at runtime
Kernel config	linux-raspberrypi recipe in meta-raspberrypi; config fragments via .bbappend
PREEMPT_RT support	Via kernel config fragment in recipe append
Read-only rootfs	IMAGE_FEATURES += "read-only-rootfs" built-in

Verdict for this project: Rejected.

Yocto is the industry standard for complex embedded Linux products where reproducibility, long-term maintenance, and multi-target builds matter. However, for a single-developer audio project targeting one board (RPi4B), the overhead is disproportionate: 50GB build disk, 2-4 hour clean build, steep BitBake/recipe learning curve, and meta-layer dependency management. The project has no need for Yocto's multi-machine, multi-distro, SDK-generation capabilities. Revisit Yocto only if this project scales to a product line with multiple hardware targets.

1.3 Raspberry Pi OS Lite (64-bit, Bookworm)

Aspect	Assessment
Version	Bookworm (Debian 12), kernel 6.6 or 6.12 (rpi-update)
Architecture	aarch64 (64-bit ARMv8-A)
Install size	~3 GB (Lite, no desktop)
RAM usage (idle)	~150 MB
Init system	systemd
Package format	.deb via apt — full Debian repositories
Kernel source	github.com/raspberrypi/linux — rpi-6.12.y branch
PREEMPT_RT support	Not in stock kernel; build custom kernel with CONFIG_PREEMPT_RT=y
Read-only rootfs	Manual setup via OverlayFS or fstab ro + tmpfs mounts
Cross-compilation	aarch64-linux-gnu- toolchain (available via apt on Ubuntu/Debian host)
Maintenance	Active — Raspberry Pi Foundation + Debian upstream

Verdict for this project: Selected.

RPiOS Lite offers the right balance:

• Development speed: apt install for JACK, ALSA tools, debugging — minutes, not rebuilds
• Ecosystem: Full Debian Bookworm repository with up-to-date audio packages
• Familiarity: Standard Linux systemd-based system — no proprietary init or build system
• Custom kernel: Build once, drop kernel8.img onto /boot — simple and well-documented
• Production path: Can later repackage the working RPiOS setup into a Buildroot image if appliance-style deployment is desired

The 3GB install size and 150MB idle RAM are acceptable for an 8GB RPi4B. If size reduction becomes important later, the system can be trimmed (remove unnecessary packages, switch to read-only squashfs).

---

2. PREEMPT_RT Kernel Integration

2.1 Source Tree

The Raspberry Pi Foundation maintains a downstream kernel at:

https://github.com/raspberrypi/linux
Branch: rpi-6.12.y (64-bit, kernel8 build)
Defconfig: bcm2711_defconfig

As of May 2026, the 6.12.y RT patchset is mostly upstream. Only CONFIG_PREEMPT_RT=y needs to be enabled on top of bcm2711_defconfig.

2.2 Build Method: Cross-Compilation on x86 Host

Recommended approach — build the kernel on a fast x86 machine, then copy the resulting kernel image and DTBs to the Pi's /boot partition.

# Toolchain (Ubuntu 26.04)
sudo apt install gcc-aarch64-linux-gnu binutils-aarch64-linux-gnu

# Clone RPi kernel
git clone --depth=1 -b rpi-6.12.y \
  https://github.com/raspberrypi/linux.git linux-rpi
cd linux-rpi

# Apply config
KERNEL=kernel8
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcm2711_defconfig

# Enable PREEMPT_RT
scripts/config -e CONFIG_PREEMPT_RT
scripts/config -d CONFIG_PREEMPT
scripts/config -e CONFIG_HIGH_RES_TIMERS
scripts/config -e CONFIG_HZ_1000
scripts/config -e CONFIG_NO_HZ_FULL
scripts/config -e CONFIG_RCU_NOCB_CPU
scripts/config -e CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE
scripts/config --set-val CONFIG_HZ 1000
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- olddefconfig

# Build (adjust -j for your CPU count)
make -j$(nproc) ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
  Image modules dtbs

2.3 Kernel Installation on RPi

# Copy kernel
scp arch/arm64/boot/Image root@<pi-ip>:/boot/kernel8-rt.img

# Copy DTBs
scp arch/arm64/boot/dts/broadcom/*.dtb root@<pi-ip>:/boot/

# Install modules
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
  INSTALL_MOD_PATH=./modules modules_install
scp -r modules/lib/modules/* root@<pi-ip>:/lib/modules/

# On the Pi, update config.txt:
# kernel=kernel8-rt.img
# Add cmdline.txt: threadirqs nohz_full=1-3 rcu_nocbs=1-3 isolcpus=1-3

2.4 Kernel Configuration Checklist

Config	Value	Rationale
CONFIG_PREEMPT_RT	y	Full real-time preemption
CONFIG_PREEMPT	n	Disable voluntary-only preemption
CONFIG_HZ	1000	1ms timer tick for lower scheduling jitter
CONFIG_HZ_1000	y	Required for HZ=1000
CONFIG_HIGH_RES_TIMERS	y	High-resolution timer subsystem
CONFIG_NO_HZ_FULL	y	Tickless operation on isolated CPUs
CONFIG_RCU_NOCB_CPU	y	Offload RCU callbacks from RT CPUs
CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE	y	Lock CPU at max frequency
CONFIG_CPU_FREQ_GOV_PERFORMANCE	y	Performance governor available
CONFIG_IRQ_FORCED_THREADING	y	Force threaded IRQ handlers
CONFIG_PREEMPTIRQ_DELAY_TEST	n	Debug — disable for production
CONFIG_USB_XHCI_HCD	y	USB 3.0 host controller (required)
CONFIG_SND_USB_AUDIO	y	USB Audio Class driver (build as module preferred)

2.5 Alternative: Native Build on RPi4B

Building natively on the RPi4B itself is possible but slow (~2-3 hours for a full kernel build). Cross-compilation on a modern x86 machine completes in ~10-15 minutes. Recommendation: Cross-compile for development iterations, fall back to native build on the Pi only if cross-compilation toolchain issues arise.

2.6 Kernel Verification

# Verify RT preemption is active
uname -v | grep PREEMPT_RT
cat /sys/kernel/realtime          # should return "1"

# Measure worst-case latency
sudo cyclictest -t 4 -p 99 -i 200 -n -l 100000 -q
# Target: max latency < 100µs on idle system

---

3. Boot Time Optimization

3.1 Current Baseline

A stock RPiOS Lite 64-bit boots in approximately:

• Firmware/Bootloader: ~3-5 seconds
• Kernel + initrd: ~5-8 seconds
• Userspace (systemd): ~10-15 seconds
• Total: ~18-28 seconds to login prompt

Measured with systemd-analyze:

systemd-analyze          # overall boot time
systemd-analyze blame    # per-unit timing
systemd-analyze critical-chain  # dependency chain
systemd-analyze plot > boot.svg  # visual timeline

3.2 Optimization Targets

Target	Approach	Expected Saving
Disable WiFi/BT	dtoverlay=disable-wifi, dtoverlay=disable-bt in config.txt	~2-3s
Reduce kernel modules	Disable unused drivers (HDMI audio, camera, codecs)	~1-2s
Minimal initramfs	Build custom initramfs with only essential modules (usb, ext4, xhci)	~2-4s
Disable unnecessary services	systemctl disable bluetooth, avahi, triggerhappy, etc.	~3-5s
Quiet boot	quiet loglevel=3 on kernel cmdline	~1-2s (fewer console writes)
Disable swap wait	Remove swapfile or reduce swap timeout	~0-5s
Fast boot in config.txt	boot_delay=0, disable_splash=1	~0.5-1s

Target: Sub-15-second boot (from power-on to JACK running).

3.3 Services to Disable

# Audio appliance — disable non-essential services
sudo systemctl disable bluetooth
sudo systemctl disable hciuart
sudo systemctl disable avahi-daemon
sudo systemctl disable triggerhappy
sudo systemctl disable ModemManager
sudo systemctl disable polkit
sudo systemctl mask systemd-random-seed.service  # if using read-only root

3.4 Kernel Command Line

console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait
quiet loglevel=3
threadirqs nohz_full=1-3 rcu_nocbs=1-3 isolcpus=1-3
usbcore.autosuspend=-1
fsck.mode=skip

Parameter	Purpose
quiet loglevel=3	Suppress verbose kernel messages
threadirqs	Force threaded IRQ handlers (RT audio)
nohz_full=1-3	Disable timer ticks on CPU cores 1-3
rcu_nocbs=1-3	Offload RCU callbacks from audio cores
isolcpus=1-3	Isolate cores 1-3 from scheduler (reserve for JACK/DSP)
usbcore.autosuspend=-1	Disable USB autosuspend (prevents audio dropouts)
fsck.mode=skip	Skip filesystem check on boot (for speed; use only on reliable storage)

3.5 CPU Core Allocation

With 4 Cortex-A72 cores:

• Core 0: System + IRQ handling (including xHCI USB interrupts)
• Cores 1-3: JACK real-time threads + DSP processing

This is enforced via:

• isolcpus=1-3 on kernel cmdline
• taskset -c 1-3 for JACK server
• IRQ affinity: pin xHCI to CPU 0 only

---

4. Read-Only Root Filesystem Options

4.1 Why Read-Only Rootfs

For an embedded audio appliance:

• Reliability: Prevents SD card corruption from unexpected power loss
• Consistency: System state is known and reproducible
• Longevity: Reduces SD card wear (no unexpected writes)
• Safety: User cannot accidentally break the system

4.2 Approach Comparison

Approach	Complexity	SD Card Wear	Runtime Writes	Boot Impact
OverlayFS (tmpfs upper)	Low	Minimal	Lost on reboot	+1-2s
OverlayFS (persistent upper on separate partition)	Medium	Low	Persisted	+1-2s
squashfs root + tmpfs overlay	Medium	None	Lost on reboot	+0s (faster read)
ext4 ro + tmpfs for /var and /tmp	Low	Minimal	Lost on reboot	+0s
Full read-write ext4	None	Full	Normal	Baseline

4.3 Recommended: OverlayFS with Persistent /data

┌─────────────────────────────────────────┐
│  SD Card / NVMe / USB SSD               │
├──────────┬──────────┬───────────────────┤
│ /boot    │ /        │ /data             │
│ FAT32    │ ext4 ro  │ ext4 rw           │
│ 256MB    │ 4GB      │ remaining         │
│ (kernel, │ (base OS)│ (configs, presets,│
│  firmware│          │  logs, user data) │
└──────────┴──────────┴───────────────────┘

Boot process:

1. Kernel mounts /dev/mmcblk0p2 as read-only ext4
2. initramfs sets up OverlayFS:
   • Lower: /dev/mmcblk0p2 (read-only root)
   • Upper: tmpfs (writable layer, lost on reboot)
   • Workdir: tmpfs
3. /data partition mounted read-write for persistent state
4. /var and /tmp are tmpfs

To remount root read-write temporarily (for system updates):

sudo mount -o remount,rw /
# ... make changes ...
sudo mount -o remount,ro /
sudo sync

4.4 OverlayFS initramfs Setup

In /etc/initramfs-tools/scripts/init-bottom/overlay:

#!/bin/sh
# Setup OverlayFS root
mount -t tmpfs tmpfs /overlay
mkdir -p /overlay/upper /overlay/work
mount -t overlay overlay \
  -o lowerdir=${ROOT},upperdir=/overlay/upper,workdir=/overlay/work \
  ${rootmnt}

4.5 Alternative: Buildroot SquashFS Image

If an appliance-style read-only image is desired for production deployment, Buildroot can produce a squashfs rootfs (~120MB compressed) that boots with initramfs. The squashfs approach eliminates SD card writes entirely for the system partition and reduces image size. This can be explored as a Phase 2 optimization after the audio stack is validated on RPiOS Lite.

---

5. Partition Layout

5.1 Recommended Layout (RPiOS Lite)

Partition	Type	Size	Filesystem	Mount	Contents
1	Primary	256 MB	FAT32	/boot	GPU firmware, kernel8.img, config.txt, cmdline.txt, DTBs
2	Primary	8 GB	ext4 (ro)	/	Root filesystem (RPiOS Lite + audio stack)
3	Primary	20+ GB	ext4 (rw)	/data	JACK configs, DSP presets, session data, logs

5.2 Minimum vs Recommended Sizes

Component	Minimum	Recommended	Notes
/boot	128 MB	256 MB	Multiple kernel images + firmware backups
/ (root)	4 GB	8 GB	RPiOS Lite base ~3GB + audio packages (~500MB-1GB) + headroom
/data	2 GB	20+ GB	Session recordings, presets, logs (use remaining SD card space)

5.3 Storage Medium

Medium	Pros	Cons	Verdict
SD Card (32GB A2)	Cheap, no USB bus contention	Wear, slower I/O, shared bus	Acceptable for development
USB 3.0 SSD	Fast, durable, large capacity	Shares USB bus with audio interfaces!	⚠️ Risk of xrun interference
NVMe SSD (via HAT)	Fast, no USB bus contention	Requires HAT, adds cost	Best for production

Recommendation: SD card for development; NVMe HAT for production if latency-sensitive I/O is needed during operation. Avoid USB SSD as primary storage — it competes with audio interface isochronous bandwidth on the VL805 controller.

5.5 USB Boot — Ditch the SD Card

The RPi4B supports native USB boot (USB mass storage device boot mode in EEPROM). This is recommended for this project — USB SSDs and NVMe adapters are far more reliable than SD cards under sustained audio write loads.

NVMe HAT (Best Option)

Uses dedicated PCIe lanes — does not share the USB bus with audio interfaces.

Option	Pros	Cons	Cost
NVMe Base HAT (official)	Uses PCIe, no bus contention, fast	Adds height, needs spacer	~€15
Pimoroni NVMe Base	Same, well-documented	Same	~€15
Geekworm X1001 / X1002	Low profile, M.2 2230/2242	Some need EEPROM update	~€15-25

USB 3.0 SSD Boot

Booting from USB SSD is possible but contraindicated during audio operation — it competes with audio interfaces on the shared VL805 USB controller.

Aspect	Detail
Boot EEPROM setting	sudo rpi-eeprom-config --edit → BOOT_ORDER=0xf41
Performance	~350 MB/s sequential vs ~90 MB/s on SD
Runtime conflict	I/O contention with USB audio on same bus — risk of xruns
Mitigation	Boot from USB SSD, then minimize writes during audio (log to tmpfs, use OverlayFS)

Recommended Config

• Development: SD card (cheap, easy to flash)
• Production / Live use: NVMe HAT + SSD via PCIe (best reliability, no USB conflict)
• Fallback: USB SSD boot if NVMe HAT unavailable — with OverlayFS + tmpfs for runtime paths

EEPROM USB Boot Enable

# Check current boot order
sudo rpi-eeprom-config | grep BOOT_ORDER

# Set boot order: USB → SD → Restart
echo 'BOOT_ORDER=0xf41' | sudo tee -a /boot/firmware/config.txt

# Or via rpi-eeprom-config
sudo rpi-eeprom-config --edit
# Change BOOT_ORDER to 0xf241 for SD → USB → Restart
# Change BOOT_ORDER to 0xf41 for USB → SD → Restart

# Reboot to apply
sudo reboot

Boot order codes:

Mode	Value
SD card	0x1
USB mass storage	0x4
NVMe (if available)	0x6
Restart	0xf
SD → USB → Restart	0xf241
USB → SD → Restart	0xf41
NVMe → USB → SD → Restart	0xf614

Also ensure P1-R3 (build script task) includes an NVMe HAT HOWTO section.

5.4 Image Size Constraints

For SD card deployment:

• Minimum card: 16 GB (Class A2 recommended for random I/O)
• Image size: ~4 GB compressed (RPiOS Lite + audio stack)
• Expanded on first boot: Auto-expand to fill SD card

For Buildroot appliance image (future):

• Compressed image: ~50-80 MB (kernel + squashfs root)
• Install size: ~200-300 MB on disk

---

6. Cross-Compilation Toolchain

6.1 Toolchain Selection

Recommended: gcc-aarch64-linux-gnu from Ubuntu 26.04 repositories.

sudo apt install \
  gcc-aarch64-linux-gnu \
  g++-aarch64-linux-gnu \
  binutils-aarch64-linux-gnu \
  libc6-dev-arm64-cross

6.2 Toolchain Details

Component	Package	Version (Ubuntu 26.04)
C compiler	gcc-aarch64-linux-gnu	14.2.0 / 15.x
C++ compiler	g++-aarch64-linux-gnu	14.2.0 / 15.x
Binutils	binutils-aarch64-linux-gnu	2.43
C library	libc6-dev-arm64-cross	glibc 2.40
Target triple	aarch64-linux-gnu	—
Architecture	ARMv8-A (Cortex-A72)	-march=armv8-a+crc+simd

6.3 Environment Setup

export ARCH=arm64
export CROSS_COMPILE=aarch64-linux-gnu-
export KERNEL=kernel8

# Verify
aarch64-linux-gnu-gcc --version
# aarch64-linux-gnu-gcc (Ubuntu 14.2.0-4ubuntu2) 14.2.0

6.4 Kernel Build Commands

# Configure
make bcm2711_defconfig
make menuconfig   # enable PREEMPT_RT and tuning options

# Build
make -j$(nproc) Image modules dtbs

# Output files:
#   arch/arm64/boot/Image           → kernel8-rt.img (on Pi /boot)
#   arch/arm64/boot/dts/broadcom/   → *.dtb files (on Pi /boot)
#   modules (INSTALL_MOD_PATH)      → /lib/modules/<version>/ (on Pi)

6.5 Alternative: Bootlin Toolchains

Buildroot uses Bootlin pre-built toolchains. These are available standalone:

https://toolchains.bootlin.com/
→ aarch64--glibc--stable-2024.02-1.tar.bz2

Bootlin toolchains include the full sysroot (headers, libraries) needed for building userspace as well as the kernel. This is useful if building userspace audio tools from source (e.g., a custom JACK build), but for kernel-only cross-compilation, the distro aarch64-linux-gnu- package is sufficient.

6.6 Native Compilation Fallback

If cross-compilation proves problematic (32-bit host, missing headers), the RPi4B 8GB can build its own kernel natively in ~2-3 hours:

# On the Pi:
git clone --depth=1 -b rpi-6.12.y https://github.com/raspberrypi/linux
cd linux
KERNEL=kernel8
make bcm2711_defconfig
# Edit .config for PREEMPT_RT
make -j4 Image modules dtbs
sudo make modules_install
sudo cp arch/arm64/boot/Image /boot/kernel8-rt.img

---

7. Implementation Roadmap

Phase 1: Base OS Setup (this phase)

1. Flash RPiOS Lite 64-bit to SD card
2. First boot: enable SSH, set hostname, update packages
3. Disable unnecessary services (WiFi, Bluetooth, avahi, etc.)
4. Apply boot optimizations (config.txt, cmdline.txt)
5. Configure CPU isolation (isolcpus=1-3)
6. Test baseline boot time with systemd-analyze

Phase 2: RT Kernel Build (next phase, P2)

1. Set up cross-compilation toolchain on x86 host
2. Clone rpi-6.12.y kernel
3. Apply PREEMPT_RT + tuning config
4. Build kernel, modules, DTBs
5. Install on Pi and verify with cyclictest
6. Target: max latency < 100µs

Phase 3: Read-Only Rootfs (follows audio stack validation)

1. Test OverlayFS initramfs approach
2. Create /data partition for persistent state
3. Configure tmpfs for /var, /tmp, /run
4. Test update procedure (remount-rw → apt → remount-ro)

Phase 4: Production Image (optional, post-validation)

1. Evaluate Buildroot for appliance image
2. Create squashfs rootfs with initramfs
3. Generate reproducible SD card image
4. CI pipeline for kernel + rootfs builds

---

8. Risk Register

Risk	Likelihood	Impact	Mitigation
Cross-compilation toolchain mismatch (glibc version skew)	Low	Medium	Use matching toolchain; fall back to native build
PREEMPT_RT kernel causes USB/xHCI regressions	Medium	High	Test thoroughly; maintain stock kernel as fallback in /boot
SD card wear from audio logging	Medium	Low	Log to tmpfs; archive to /data periodically
USB SSD storage conflicts with audio isochronous transfers	Medium	High	Use SD card or NVMe HAT; avoid USB storage during audio operation
RPi kernel rpi-6.12.y diverges from mainline PREEMPT_RT	Low	Medium	Monitor kernel releases; test each new kernel version
Read-only rootfs breaks apt/package management	Low	Medium	Document remount-rw procedure; automate in update script

---

9. References

1. Raspberry Pi Linux kernel: https://github.com/raspberrypi/linux (branch: rpi-6.12.y)
2. Buildroot manual: https://buildroot.org/downloads/manual/manual.html
3. Buildroot RPi4 64-bit defconfig: https://github.com/buildroot/buildroot/tree/master/configs/raspberrypi4_64_defconfig
4. Yocto meta-raspberrypi: https://github.com/agherzan/meta-raspberrypi
5. Yocto Project docs: https://docs.yoctoproject.org/
6. PREEMPT_RT wiki: https://wiki.linuxfoundation.org/realtime/start
7. Bootlin toolchains: https://toolchains.bootlin.com/
8. Zynthian OS (RPi RT audio reference): https://github.com/zynthian/zynthian-sys
9. Linux kernel RT config docs: https://www.kernel.org/doc/html/latest/locking/rt-mutex-design.html
10. RPi boot options: https://www.raspberrypi.com/documentation/computers/config_txt.html#boot-options
11. OverlayFS kernel docs: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html
12. systemd-analyze: https://www.freedesktop.org/software/systemd/man/systemd-analyze.html