raspberry-pi-mixer/docs/troubleshooting.md

Troubleshooting Guide

Common issues and solutions for the Raspberry Pi Real-Time Audio Mixer.

Table of Contents

1. Audio Issues
2. MIDI Issues
3. Network & Web UI Issues
4. Recording Issues
5. Streaming Issues
6. System Issues
7. Build Issues
8. Diagnostic Tools

---

1. Audio Issues

No audio output

Symptoms: Everything looks normal but no sound comes out.

Checklist:

# 1. Is the USB audio interface detected?
aplay -l
arecord -l
cat /proc/asound/cards

# 2. Is JACK running?
systemctl status jackd
ps aux | grep jackd

# 3. Are ports connected?
jack_lsp -c

# 4. Is the channel unmuted and volume up?
curl -s http://localhost:8080/channels/1 \
     -H "X-API-Key: your-key" | jq '.mute, .volume'

# 5. Are system outputs unmuted?
alsamixer    # Press F6 to select USB device, check levels

Common fixes:

• sudo systemctl restart jackd && sleep 2 && sudo systemctl restart mixer-api
• Check alsamixer — some interfaces have hardware mute switches
• Verify cables: input on capture_1 not playback_1
• If using headphones, check the headphone output routing

---

xruns (audio dropouts, clicks, pops)

Symptoms: Audio glitches, crackling, periodic dropouts, JACK log shows XRUN.

Check JACK status:

# View JACK logs for xrun count
journalctl -u jackd -n 50 | grep -i xrun

# Check CPU isolation
cat /sys/devices/system/cpu/isolated            # should show 1-3
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor  # should be 'performance'

Troubleshooting order:

1. Increase buffer size. Edit /etc/systemd/system/jackd.service:

   -p 128 → -p 256 (or -p 512 for extreme cases)
   

   Then: sudo systemctl daemon-reload && sudo systemctl restart jackd

2. Check CPU governor:

   cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
   # All should say 'performance'. If not:
   sudo systemctl restart cpu-performance
   

3. Verify RT priorities:

   ps -eo pid,cls,rtprio,comm | grep -E 'jackd|irq'
   # jackd should have CLS=FF (SCHED_FIFO), RTPRIO=70
   # xhci IRQ thread should have RTPRIO=95
   

4. Check for USB bandwidth issues:

   lsusb -t    # Check which devices share the USB bus
   # Move non-audio USB devices to a different hub/port
   

5. Reduce channel count or effects:

   • Fewer active channels → less DSP load
   • Disable unused plugins
   • Reduce EQ bands (or disable EQ on unused channels)

6. Check real-time group membership:

   groups | grep audio     # must include 'audio'
   ulimit -r               # should show 99
   ulimit -l               # should show unlimited
   

---

Audio interface not detected

Symptoms: aplay -l doesn't show the USB interface.

Checklist:

# Is the device visible to USB?
lsusb
# Look for your audio interface in the list

# Kernel messages about USB audio
dmesg | grep -i 'usb\|audio\|snd' | tail -30

# Are ALSA modules loaded?
lsmod | grep snd_usb

Fixes:

• Try a different USB port (USB 3.0 blue ports preferred)
• Try a different USB cable (short, shielded cables work best)
• Power-cycle the audio interface
• If bus-powered, use a powered USB hub
• Check the interface is in class-compliant mode (not proprietary driver mode):
  • Focusrite: hold 48V button while connecting USB (Gen 3 only)
  • Behringer: generally always class-compliant

---

USB audio interface randomly disconnects

Symptoms: Interface disappears from aplay -l mid-session, JACK crashes.

Causes and fixes:

1. Under-voltage — check for lightning bolt on HDMI display or run:

   vcgencmd get_throttled
   # 0x0 = no issues. 0x50000 = under-voltage has occurred
   

   Fix: Use official RPi PSU (5.1V/3A), shorter/better USB-C cable.

2. USB cable — long or poor-quality USB cables cause disconnects. Fix: Use a cable ≤2m, USB-IF certified.

3. Power saving — USB autosuspend may disconnect the interface:

   # Check current setting
   cat /sys/module/usbcore/parameters/autosuspend
   # Disable autosuspend (-1 = never)
   echo -1 | sudo tee /sys/module/usbcore/parameters/autosuspend
   

4. EMI interference — nearby power supplies, motors, or RF sources. Fix: Use ferrite chokes on USB cables, relocate away from interference.

---

High latency (round-trip >20ms)

# Measure round-trip latency
jack_delay

# Check current buffer settings
jack_bufsize

To reduce latency:

1. Reduce buffer size in /etc/systemd/system/jackd.service: -p 128 → -p 64 (more aggressive, may cause xruns)
2. Reduce periods: -n 3 → -n 2
3. Ensure PREEMPT_RT kernel is loaded:

   uname -r   # should contain 'rt' or 'PREEMPT_RT'
   

4. Verify kernel cmdline tuning took effect:

   cat /proc/cmdline   # should show isolcpus, threadirqs, etc.
   

---

2. MIDI Issues

MIDI controller not detected

# List MIDI devices
aconnect -l
amidi -l

# Check USB detection
lsusb | grep -i midi

# Check kernel MIDI driver
dmesg | grep -i midi

Fixes:

• Try a different USB port
• Some controllers need to be in a specific mode (e.g., Akai MIDImix: hold "SEND ALL" while connecting)
• Ensure snd-seq and snd-usb-audio modules are loaded:

  sudo modprobe snd-seq
  sudo modprobe snd-usb-audio
  

---

MIDI controller not controlling parameters

# Monitor raw MIDI events
aseqdump -p 20:0   # replace with correct port number from aconnect -l

# Check if MIDI engine is running
systemctl status mixer-api
journalctl -u mixer-api | grep -i midi

# Check if mappings exist
curl http://localhost:8080/midi/mappings \
     -H "X-API-Key: your-key"

Fixes:

• Use MIDI Learn to map controls:

  curl -X POST http://pi-mixer.local:8080/midi/learn/start \
       -H "X-API-Key: your-key"
  # Click parameter in web UI, then move physical control
  curl -X POST http://pi-mixer.local:8080/midi/learn/stop \
       -H "X-API-Key: your-key"
  

• Check the controller is sending CC messages (not SysEx or NRPN only)
• Verify the MIDI channel matches your mapping

---

MIDI Clock not syncing

# Check clock mode
curl http://localhost:8080/midi/clock \
     -H "X-API-Key: your-key"

# Set clock mode explicitly
curl -X PUT http://localhost:8080/midi/clock/mode \
     -H "X-API-Key: your-key" \
     -H "Content-Type: application/json" \
     -d '{"mode": "slave"}'

---

3. Network & Web UI Issues

Web UI not accessible

Checklist:

# Is the mixer API running?
systemctl status mixer-api
curl http://localhost:8080/stats -H "X-API-Key: your-key"

# What's the Pi's IP?
ip addr show | grep "inet "

# Is the port open?
ss -tlnp | grep 8080

# Can you reach it from another machine?
ping pi-mixer.local
curl http://pi-mixer.local:8080/stats -H "X-API-Key: your-key"

Fixes:

• Restart the API: sudo systemctl restart mixer-api
• Check the service logs: journalctl -u mixer-api -n 50
• If .local doesn't resolve, use the IP address directly
• Check firewall: sudo iptables -L (port 8080 must be open)
• For WiFi, ensure the Pi and your device are on the same network

---

WebSocket disconnects frequently

Symptoms: Web UI shows "Disconnected" banner, reconnects intermittently.

Fixes:

• Use Ethernet instead of WiFi
• Reduce browser tab count (WebSocket connections consume memory)
• Check API server logs: journalctl -u mixer-api -f
• Increase WebSocket ping interval in src/network/websocket.py

---

API returns "Unauthorized"

Check API key:

# Find your API key
grep API_KEY /etc/systemd/system/mixer-api.service

# Test with the key
curl http://localhost:8080/stats -H "X-API-Key: your-api-key"

To reset the API key:

# Generate a new key
NEW_KEY=$(openssl rand -hex 16)
echo "Generated key: $NEW_KEY"

# Update the service file
sudo sed -i "s/Environment=MIXER_API_KEY=.*/Environment=MIXER_API_KEY=$NEW_KEY/" \
    /etc/systemd/system/mixer-api.service

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart mixer-api

---

Cannot connect via SSH

# Check SSH service
sudo systemctl status ssh

# Enable if disabled
sudo systemctl enable --now ssh

# Find the IP
ip addr show | grep "inet " | grep -v 127.0.0.1

Default SSH credentials: pi / raspberry. Change immediately:

passwd

---

4. Recording Issues

Recording fails to start

Checklist:

# Is the data partition mounted?
df -h /data

# Is there enough space?
df -h /data | awk 'NR==2 {print $4}'  # available space

# Are channels armed?
curl http://localhost:8080/recording/status \
     -H "X-API-Key: your-key"

Fixes:

• Arm at least one channel before recording
• Ensure /data partition exists and is writable:

  sudo mkdir -p /data/recordings
  sudo chown pi:pi /data/recordings
  

• Free up space: delete old recordings in /data/recordings/

---

Recorded files are corrupt or silent

Check:

# Check WAV file validity
ffprobe /data/recordings/session_001/channel_01.wav

# Check file size (non-zero?)
ls -lh /data/recordings/session_001/

Causes:

• JACK wasn't running or crashed during recording → restart JACK and re-record
• SD card too slow → use A2-class card
• Disk full mid-recording → check disk space before long sessions
• Channel not armed → arm channel before pressing Record

---

SD card too slow for multi-track recording

# Test write speed
dd if=/dev/zero of=/data/test_write bs=1M count=256 conv=fdatasync 2>&1 | \
    tail -1 | awk '{print $8 " " $9}'
# Should show >20 MB/s for 16-track recording

# Check SD card class
sudo cat /sys/block/mmcblk0/device/cid
# Use the CID to look up the card specs

Fixes:

• Use A2-class SD card (SanDisk Extreme Pro or Samsung EVO Select)
• Reduce track count (16→8 channels)
• Reduce bit depth (32-bit float→24-bit or 16-bit)
• Use external USB SSD for recording:

  # Mount USB drive and configure recording path
  sudo mount /dev/sda1 /mnt/recordings
  curl -X PUT http://localhost:8080/recording/path \
       -H "X-API-Key: your-key" \
       -d '{"path": "/mnt/recordings"}'
  

---

5. Streaming Issues

Stream won't start

Checklist:

# Is GStreamer installed?
which gst-launch-1.0
gst-launch-1.0 --version

# Is a camera connected?
v4l2-ctl --list-devices

# Can you reach the streaming server?
curl -s --max-time 5 rtmp://a.rtmp.youtube.com/live2/ 2>&1
# Should return RTMP handshake, not timeout

Fixes:

• Install GStreamer: sudo apt install gstreamer1.0-tools gstreamer1.0-plugins-*
• Verify stream key is correct (no extra spaces)
• Check internet connection: ping 8.8.8.8
• Some networks block RTMP — try on a different network
• Reduce bitrate: 4500 → 2500 Kbps for slow connections

---

Stream is choppy or dropping frames

# Monitor stream stats
curl http://localhost:8080/stream/stats \
     -H "X-API-Key: your-key"

Fixes:

• Use Ethernet — WiFi is unreliable for streaming
• Reduce video bitrate (4500→2500 Kbps) and resolution (1080p→720p)
• Reduce audio channels in the stream mix
• Close other network-intensive applications
• Check CPU usage: top — streaming should be <50% CPU
• Use hardware-accelerated encoding if available (omx264 instead of x264)

---

Camera not detected

# List video devices
v4l2-ctl --list-devices
ls /dev/video*

# Check Pi Camera
vcgencmd get_camera

Fixes:

• Pi Camera: ensure ribbon cable is properly seated
• USB Camera: try a different USB port
• Check camera is supported: v4l2-ctl --list-formats-ext
• Enable Pi Camera in raspi-config → Interface Options → Camera

---

6. System Issues

Pi won't boot

Checklist:

1. Power LED — solid red means power OK. Flashing red = under-voltage.
2. ACT LED — green activity light should flash. No flashing = SD card issue.
3. HDMI display — connect a monitor to see boot messages.

Fixes:

• Re-flash the SD card
• Try a different power supply (5V/3A)
• Check the SD card is properly inserted
• Verify the image was written correctly:

  # Compare checksums
  sha256sum build/out/rpi-audio-mixer-*.img.xz
  

---

Pi is slow or unresponsive

# Check CPU temperature
vcgencmd measure_temp

# Check CPU frequency
vcgencmd measure_clock arm

# Check throttling status
vcgencmd get_throttled
# 0x0 = OK
# 0x50000 = under-voltage
# 0x50005 = throttled (temperature >80°C)

Fixes:

• Add a heatsink to the CPU
• Ensure proper ventilation
• Reduce channel count or effects
• Check for background processes: top

---

First-boot wizard keeps appearing

# Check if re-arm flag exists
ls -la /force-firstboot

# Remove it
sudo rm /force-firstboot

# Disable the first-boot service
sudo systemctl disable mixer-firstboot

---

Systemd services fail to start

# Check overall service status
systemctl --failed

# Check specific service
journalctl -u mixer-api -n 100

# Check dependencies
systemctl list-dependencies mixer-api

# Verify service files exist
ls -la /etc/systemd/system/mixer-*.service

Common fixes:

• JACK must start before mixer-api:

  sudo systemctl start jackd
  sleep 2
  sudo systemctl start mixer-api
  

• Permissions issue: ensure pi user is in the audio group
• Config file syntax error: systemd-analyze verify /etc/systemd/system/mixer-api.service

---

Disk full

# Check disk usage
df -h

# Find large directories
du -sh /data/* | sort -rh | head -10

# Clean up
sudo journalctl --vacuum-size=100M     # limit journal size
rm -rf /data/recordings/old_session/  # delete old recordings

---

7. Build Issues

Build fails — see build/README.md for common issues

Common build problems:

Problem	Solution
qemu-aarch64-static not found	sudo apt install qemu-user-static && sudo update-binfmts --enable qemu-aarch64
apt fails in chroot	DNS not resolving. Check host /etc/resolv.conf.
No space left on device	Increase image size: --image-size 16384
kernel build fails	Install deps: sudo apt install gcc-aarch64-linux-gnu flex bison libssl-dev bc
loop device busy	sudo losetup -D to detach stale loop devices
permission denied	Build requires sudo. Ensure passwordless sudo is configured.

---

8. Diagnostic Tools

Quick Health Check

#!/bin/bash
# mixer-health-check.sh — run on the Pi for a quick diagnostic

echo "=== Mixer Health Check ==="
echo

echo "--- Kernel ---"
uname -r | grep -q rt && echo "✅ PREEMPT_RT kernel" || echo "❌ Stock kernel (no RT)"
echo

echo "--- Audio ---"
aplay -l 2>/dev/null | grep -q USB && echo "✅ USB audio interface detected" || echo "❌ No USB audio"
echo

echo "--- Services ---"
for svc in jackd mixer-api mixer-ui; do
    systemctl is-active --quiet $svc && echo "✅ $svc running" || echo "❌ $svc not running"
done
echo

echo "--- CPU ---"
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null
echo "CPU temp: $(vcgencmd measure_temp)"
echo

echo "--- Disk ---"
df -h / /data 2>/dev/null
echo

echo "--- Network ---"
ip -4 addr show | grep "inet " | grep -v 127.0.0.1
echo

echo "--- Recent Errors ---"
journalctl -p err -n 5 --no-pager

Useful Diagnostic Commands

# Audio
aplay -l                                # List playback devices
arecord -l                              # List capture devices
jack_lsp -c                             # JACK ports and connections
jack_bufsize                            # Current JACK buffer size
cat /proc/asound/cards                  # Kernel audio device list
amidi -l                                # List MIDI devices

# System
vcgencmd get_throttled                  # Throttling/voltage status
vcgencmd measure_temp                   # CPU temperature
cat /proc/cmdline                       # Kernel boot parameters
cat /sys/devices/system/cpu/isolated    # CPU isolation
ulimit -r                               # Max real-time priority
groups                                  # User group membership

# USB
lsusb -t                                # USB device tree (bandwidth)
lsusb -v 2>/dev/null | grep -i audio    # Audio device details

# Services
systemctl --failed                      # Failed services
journalctl -u mixer-api -n 50           # API server logs
journalctl -u jackd -n 50               # JACK logs

# Network
ip addr show                            # IP addresses
ss -tlnp                                # Listening ports
curl localhost:8080/stats               # API health check

Getting Help

1. Run the health check script above
2. Collect logs:

   journalctl -u jackd -u mixer-api -u mixer-ui --since "10 minutes ago" > mixer-logs.txt
   

3. Include dmesg | tail -50 output
4. Report hardware details: cat /proc/cpuinfo | grep Model, lsusb

---

Appendix: JACK Error Messages Decoded

Error	Meaning	Fix
Cannot lock down memory	User lacks memlock limits	Add user to audio group, check /etc/security/limits.d/99-audio.conf
Cannot use real-time scheduling	User lacks rtprio limits	Same as above
XRUN callback	Buffer underrun/overrun	Increase buffer size, check CPU governor
Cannot connect to server socket	JACK server not running	sudo systemctl start jackd
ALSA: cannot open audio device hw:USB	Device busy or wrong name	Check aplay -l for correct device name
Cannot create new client	Too many JACK clients	Close unused clients, increase client limit
Cannot allocate memory	System out of memory	Reduce buffers, close other applications
subgraph starting at ... timed out	A JACK client is unresponsive	Restart the stuck client