# Troubleshooting Guide

Common issues and solutions for the Raspberry Pi Real-Time Audio Mixer.

## Table of Contents

1. [Audio Issues](#1-audio-issues)
2. [MIDI Issues](#2-midi-issues)
3. [Network & Web UI Issues](#3-network--web-ui-issues)
4. [Recording Issues](#4-recording-issues)
5. [Streaming Issues](#5-streaming-issues)
6. [System Issues](#6-system-issues)
7. [Build Issues](#7-build-issues)
8. [Diagnostic Tools](#8-diagnostic-tools)

---

## 1. Audio Issues

### No audio output

**Symptoms:** Everything looks normal but no sound comes out.

**Checklist:**
```bash
# 1. Is the USB audio interface detected?
aplay -l
arecord -l
cat /proc/asound/cards

# 2. Is JACK running?
systemctl status jackd
ps aux | grep jackd

# 3. Are ports connected?
jack_lsp -c

# 4. Is the channel unmuted and volume up?
curl -s http://localhost:8080/channels/1 \
     -H "X-API-Key: your-key" | jq '.mute, .volume'

# 5. Are system outputs unmuted?
alsamixer    # Press F6 to select USB device, check levels
```

**Common fixes:**
- `sudo systemctl restart jackd && sleep 2 && sudo systemctl restart mixer-api`
- Check `alsamixer` — some interfaces have hardware mute switches
- Verify cables: input on capture_1 not playback_1
- If using headphones, check the headphone output routing

---

### xruns (audio dropouts, clicks, pops)

**Symptoms:** Audio glitches, crackling, periodic dropouts, JACK log shows `XRUN`.

**Check JACK status:**
```bash
# View JACK logs for xrun count
journalctl -u jackd -n 50 | grep -i xrun

# Check CPU isolation
cat /sys/devices/system/cpu/isolated            # should show 1-3
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor  # should be 'performance'
```

**Troubleshooting order:**

1. **Increase buffer size.** Edit `/etc/systemd/system/jackd.service`:
   ```
   -p 128 → -p 256 (or -p 512 for extreme cases)
   ```
   Then: `sudo systemctl daemon-reload && sudo systemctl restart jackd`

2. **Check CPU governor:**
   ```bash
   cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
   # All should say 'performance'. If not:
   sudo systemctl restart cpu-performance
   ```

3. **Verify RT priorities:**
   ```bash
   ps -eo pid,cls,rtprio,comm | grep -E 'jackd|irq'
   # jackd should have CLS=FF (SCHED_FIFO), RTPRIO=70
   # xhci IRQ thread should have RTPRIO=95
   ```

4. **Check for USB bandwidth issues:**
   ```bash
   lsusb -t    # Check which devices share the USB bus
   # Move non-audio USB devices to a different hub/port
   ```

5. **Reduce channel count or effects:**
   - Fewer active channels → less DSP load
   - Disable unused plugins
   - Reduce EQ bands (or disable EQ on unused channels)

6. **Check real-time group membership:**
   ```bash
   groups | grep audio     # must include 'audio'
   ulimit -r               # should show 99
   ulimit -l               # should show unlimited
   ```

---

### Audio interface not detected

**Symptoms:** `aplay -l` doesn't show the USB interface.

**Checklist:**
```bash
# Is the device visible to USB?
lsusb
# Look for your audio interface in the list

# Kernel messages about USB audio
dmesg | grep -i 'usb\|audio\|snd' | tail -30

# Are ALSA modules loaded?
lsmod | grep snd_usb
```

**Fixes:**
- Try a different USB port (USB 3.0 blue ports preferred)
- Try a different USB cable (short, shielded cables work best)
- Power-cycle the audio interface
- If bus-powered, use a powered USB hub
- Check the interface is in class-compliant mode (not proprietary driver mode):
  - Focusrite: hold 48V button while connecting USB (Gen 3 only)
  - Behringer: generally always class-compliant

---

### USB audio interface randomly disconnects

**Symptoms:** Interface disappears from `aplay -l` mid-session, JACK crashes.

**Causes and fixes:**

1. **Under-voltage** — check for lightning bolt on HDMI display or run:
   ```bash
   vcgencmd get_throttled
   # 0x0 = no issues. 0x50000 = under-voltage has occurred
   ```
   Fix: Use official RPi PSU (5.1V/3A), shorter/better USB-C cable.

2. **USB cable** — long or poor-quality USB cables cause disconnects.
   Fix: Use a cable ≤2m, USB-IF certified.

3. **Power saving** — USB autosuspend may disconnect the interface:
   ```bash
   # Check current setting
   cat /sys/module/usbcore/parameters/autosuspend
   # Disable autosuspend (-1 = never)
   echo -1 | sudo tee /sys/module/usbcore/parameters/autosuspend
   ```

4. **EMI interference** — nearby power supplies, motors, or RF sources.
   Fix: Use ferrite chokes on USB cables, relocate away from interference.

---

### High latency (round-trip >20ms)

```bash
# Measure round-trip latency
jack_delay

# Check current buffer settings
jack_bufsize
```

**To reduce latency:**
1. Reduce buffer size in `/etc/systemd/system/jackd.service`:
   `-p 128` → `-p 64` (more aggressive, may cause xruns)
2. Reduce periods: `-n 3` → `-n 2`
3. Ensure PREEMPT_RT kernel is loaded:
   ```bash
   uname -r   # should contain 'rt' or 'PREEMPT_RT'
   ```
4. Verify kernel cmdline tuning took effect:
   ```bash
   cat /proc/cmdline   # should show isolcpus, threadirqs, etc.
   ```

---

## 2. MIDI Issues

### MIDI controller not detected

```bash
# List MIDI devices
aconnect -l
amidi -l

# Check USB detection
lsusb | grep -i midi

# Check kernel MIDI driver
dmesg | grep -i midi
```

**Fixes:**
- Try a different USB port
- Some controllers need to be in a specific mode (e.g., Akai MIDImix:
  hold "SEND ALL" while connecting)
- Ensure `snd-seq` and `snd-usb-audio` modules are loaded:
  ```bash
  sudo modprobe snd-seq
  sudo modprobe snd-usb-audio
  ```

---

### MIDI controller not controlling parameters

```bash
# Monitor raw MIDI events
aseqdump -p 20:0   # replace with correct port number from aconnect -l

# Check if MIDI engine is running
systemctl status mixer-api
journalctl -u mixer-api | grep -i midi

# Check if mappings exist
curl http://localhost:8080/midi/mappings \
     -H "X-API-Key: your-key"
```

**Fixes:**
- Use **MIDI Learn** to map controls:
  ```bash
  curl -X POST http://pi-mixer.local:8080/midi/learn/start \
       -H "X-API-Key: your-key"
  # Click parameter in web UI, then move physical control
  curl -X POST http://pi-mixer.local:8080/midi/learn/stop \
       -H "X-API-Key: your-key"
  ```
- Check the controller is sending CC messages (not SysEx or NRPN only)
- Verify the MIDI channel matches your mapping

---

### MIDI Clock not syncing

```bash
# Check clock mode
curl http://localhost:8080/midi/clock \
     -H "X-API-Key: your-key"

# Set clock mode explicitly
curl -X PUT http://localhost:8080/midi/clock/mode \
     -H "X-API-Key: your-key" \
     -H "Content-Type: application/json" \
     -d '{"mode": "slave"}'
```

---

## 3. Network & Web UI Issues

### Web UI not accessible

**Checklist:**
```bash
# Is the mixer API running?
systemctl status mixer-api
curl http://localhost:8080/stats -H "X-API-Key: your-key"

# What's the Pi's IP?
ip addr show | grep "inet "

# Is the port open?
ss -tlnp | grep 8080

# Can you reach it from another machine?
ping pi-mixer.local
curl http://pi-mixer.local:8080/stats -H "X-API-Key: your-key"
```

**Fixes:**
- Restart the API: `sudo systemctl restart mixer-api`
- Check the service logs: `journalctl -u mixer-api -n 50`
- If `.local` doesn't resolve, use the IP address directly
- Check firewall: `sudo iptables -L` (port 8080 must be open)
- For WiFi, ensure the Pi and your device are on the same network

---

### WebSocket disconnects frequently

**Symptoms:** Web UI shows "Disconnected" banner, reconnects intermittently.

**Fixes:**
- Use Ethernet instead of WiFi
- Reduce browser tab count (WebSocket connections consume memory)
- Check API server logs: `journalctl -u mixer-api -f`
- Increase WebSocket ping interval in `src/network/websocket.py`

---

### API returns "Unauthorized"

**Check API key:**
```bash
# Find your API key
grep API_KEY /etc/systemd/system/mixer-api.service

# Test with the key
curl http://localhost:8080/stats -H "X-API-Key: your-api-key"
```

**To reset the API key:**
```bash
# Generate a new key
NEW_KEY=$(openssl rand -hex 16)
echo "Generated key: $NEW_KEY"

# Update the service file
sudo sed -i "s/Environment=MIXER_API_KEY=.*/Environment=MIXER_API_KEY=$NEW_KEY/" \
    /etc/systemd/system/mixer-api.service

# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart mixer-api
```

---

### Cannot connect via SSH

```bash
# Check SSH service
sudo systemctl status ssh

# Enable if disabled
sudo systemctl enable --now ssh

# Find the IP
ip addr show | grep "inet " | grep -v 127.0.0.1
```

Default SSH credentials: `pi` / `raspberry`. Change immediately:
```bash
passwd
```

---

## 4. Recording Issues

### Recording fails to start

**Checklist:**
```bash
# Is the data partition mounted?
df -h /data

# Is there enough space?
df -h /data | awk 'NR==2 {print $4}'  # available space

# Are channels armed?
curl http://localhost:8080/recording/status \
     -H "X-API-Key: your-key"
```

**Fixes:**
- Arm at least one channel before recording
- Ensure `/data` partition exists and is writable:
  ```bash
  sudo mkdir -p /data/recordings
  sudo chown pi:pi /data/recordings
  ```
- Free up space: delete old recordings in `/data/recordings/`

---

### Recorded files are corrupt or silent

**Check:**
```bash
# Check WAV file validity
ffprobe /data/recordings/session_001/channel_01.wav

# Check file size (non-zero?)
ls -lh /data/recordings/session_001/
```

**Causes:**
- JACK wasn't running or crashed during recording → restart JACK and re-record
- SD card too slow → use A2-class card
- Disk full mid-recording → check disk space before long sessions
- Channel not armed → arm channel before pressing Record

---

### SD card too slow for multi-track recording

```bash
# Test write speed
dd if=/dev/zero of=/data/test_write bs=1M count=256 conv=fdatasync 2>&1 | \
    tail -1 | awk '{print $8 " " $9}'
# Should show >20 MB/s for 16-track recording

# Check SD card class
sudo cat /sys/block/mmcblk0/device/cid
# Use the CID to look up the card specs
```

**Fixes:**
- Use A2-class SD card (SanDisk Extreme Pro or Samsung EVO Select)
- Reduce track count (16→8 channels)
- Reduce bit depth (32-bit float→24-bit or 16-bit)
- Use external USB SSD for recording:
  ```bash
  # Mount USB drive and configure recording path
  sudo mount /dev/sda1 /mnt/recordings
  curl -X PUT http://localhost:8080/recording/path \
       -H "X-API-Key: your-key" \
       -d '{"path": "/mnt/recordings"}'
  ```

---

## 5. Streaming Issues

### Stream won't start

**Checklist:**
```bash
# Is GStreamer installed?
which gst-launch-1.0
gst-launch-1.0 --version

# Is a camera connected?
v4l2-ctl --list-devices

# Can you reach the streaming server?
curl -s --max-time 5 rtmp://a.rtmp.youtube.com/live2/ 2>&1
# Should return RTMP handshake, not timeout
```

**Fixes:**
- Install GStreamer: `sudo apt install gstreamer1.0-tools gstreamer1.0-plugins-*`
- Verify stream key is correct (no extra spaces)
- Check internet connection: `ping 8.8.8.8`
- Some networks block RTMP — try on a different network
- Reduce bitrate: 4500 → 2500 Kbps for slow connections

---

### Stream is choppy or dropping frames

```bash
# Monitor stream stats
curl http://localhost:8080/stream/stats \
     -H "X-API-Key: your-key"
```

**Fixes:**
- **Use Ethernet** — WiFi is unreliable for streaming
- Reduce video bitrate (4500→2500 Kbps) and resolution (1080p→720p)
- Reduce audio channels in the stream mix
- Close other network-intensive applications
- Check CPU usage: `top` — streaming should be <50% CPU
- Use hardware-accelerated encoding if available (omx264 instead of x264)

---

### Camera not detected

```bash
# List video devices
v4l2-ctl --list-devices
ls /dev/video*

# Check Pi Camera
vcgencmd get_camera
```

**Fixes:**
- Pi Camera: ensure ribbon cable is properly seated
- USB Camera: try a different USB port
- Check camera is supported: `v4l2-ctl --list-formats-ext`
- Enable Pi Camera in `raspi-config` → Interface Options → Camera

---

## 6. System Issues

### Pi won't boot

**Checklist:**
1. **Power LED** — solid red means power OK. Flashing red = under-voltage.
2. **ACT LED** — green activity light should flash. No flashing = SD card issue.
3. **HDMI display** — connect a monitor to see boot messages.

**Fixes:**
- Re-flash the SD card
- Try a different power supply (5V/3A)
- Check the SD card is properly inserted
- Verify the image was written correctly:
  ```bash
  # Compare checksums
  sha256sum build/out/rpi-audio-mixer-*.img.xz
  ```

---

### Pi is slow or unresponsive

```bash
# Check CPU temperature
vcgencmd measure_temp

# Check CPU frequency
vcgencmd measure_clock arm

# Check throttling status
vcgencmd get_throttled
# 0x0 = OK
# 0x50000 = under-voltage
# 0x50005 = throttled (temperature >80°C)
```

**Fixes:**
- Add a heatsink to the CPU
- Ensure proper ventilation
- Reduce channel count or effects
- Check for background processes: `top`

---

### First-boot wizard keeps appearing

```bash
# Check if re-arm flag exists
ls -la /force-firstboot

# Remove it
sudo rm /force-firstboot

# Disable the first-boot service
sudo systemctl disable mixer-firstboot
```

---

### Systemd services fail to start

```bash
# Check overall service status
systemctl --failed

# Check specific service
journalctl -u mixer-api -n 100

# Check dependencies
systemctl list-dependencies mixer-api

# Verify service files exist
ls -la /etc/systemd/system/mixer-*.service
```

**Common fixes:**
- JACK must start before `mixer-api`:
  ```bash
  sudo systemctl start jackd
  sleep 2
  sudo systemctl start mixer-api
  ```
- Permissions issue: ensure `pi` user is in the `audio` group
- Config file syntax error: `systemd-analyze verify /etc/systemd/system/mixer-api.service`

---

### Disk full

```bash
# Check disk usage
df -h

# Find large directories
du -sh /data/* | sort -rh | head -10

# Clean up
sudo journalctl --vacuum-size=100M     # limit journal size
rm -rf /data/recordings/old_session/  # delete old recordings
```

---

## 7. Build Issues

### Build fails — see build/README.md for common issues

Common build problems:

| Problem | Solution |
|---------|----------|
| `qemu-aarch64-static not found` | `sudo apt install qemu-user-static && sudo update-binfmts --enable qemu-aarch64` |
| `apt fails in chroot` | DNS not resolving. Check host `/etc/resolv.conf`. |
| `No space left on device` | Increase image size: `--image-size 16384` |
| `kernel build fails` | Install deps: `sudo apt install gcc-aarch64-linux-gnu flex bison libssl-dev bc` |
| `loop device busy` | `sudo losetup -D` to detach stale loop devices |
| `permission denied` | Build requires sudo. Ensure passwordless sudo is configured. |

---

## 8. Diagnostic Tools

### Quick Health Check

```bash
#!/bin/bash
# mixer-health-check.sh — run on the Pi for a quick diagnostic

echo "=== Mixer Health Check ==="
echo

echo "--- Kernel ---"
uname -r | grep -q rt && echo "✅ PREEMPT_RT kernel" || echo "❌ Stock kernel (no RT)"
echo

echo "--- Audio ---"
aplay -l 2>/dev/null | grep -q USB && echo "✅ USB audio interface detected" || echo "❌ No USB audio"
echo

echo "--- Services ---"
for svc in jackd mixer-api mixer-ui; do
    systemctl is-active --quiet $svc && echo "✅ $svc running" || echo "❌ $svc not running"
done
echo

echo "--- CPU ---"
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null
echo "CPU temp: $(vcgencmd measure_temp)"
echo

echo "--- Disk ---"
df -h / /data 2>/dev/null
echo

echo "--- Network ---"
ip -4 addr show | grep "inet " | grep -v 127.0.0.1
echo

echo "--- Recent Errors ---"
journalctl -p err -n 5 --no-pager
```

### Useful Diagnostic Commands

```bash
# Audio
aplay -l                                # List playback devices
arecord -l                              # List capture devices
jack_lsp -c                             # JACK ports and connections
jack_bufsize                            # Current JACK buffer size
cat /proc/asound/cards                  # Kernel audio device list
amidi -l                                # List MIDI devices

# System
vcgencmd get_throttled                  # Throttling/voltage status
vcgencmd measure_temp                   # CPU temperature
cat /proc/cmdline                       # Kernel boot parameters
cat /sys/devices/system/cpu/isolated    # CPU isolation
ulimit -r                               # Max real-time priority
groups                                  # User group membership

# USB
lsusb -t                                # USB device tree (bandwidth)
lsusb -v 2>/dev/null | grep -i audio    # Audio device details

# Services
systemctl --failed                      # Failed services
journalctl -u mixer-api -n 50           # API server logs
journalctl -u jackd -n 50               # JACK logs

# Network
ip addr show                            # IP addresses
ss -tlnp                                # Listening ports
curl localhost:8080/stats               # API health check
```

### Getting Help

1. Run the health check script above
2. Collect logs:
   ```bash
   journalctl -u jackd -u mixer-api -u mixer-ui --since "10 minutes ago" > mixer-logs.txt
   ```
3. Include `dmesg | tail -50` output
4. Report hardware details: `cat /proc/cpuinfo | grep Model`, `lsusb`

---

## Appendix: JACK Error Messages Decoded

| Error | Meaning | Fix |
|-------|---------|-----|
| `Cannot lock down memory` | User lacks `memlock` limits | Add user to `audio` group, check `/etc/security/limits.d/99-audio.conf` |
| `Cannot use real-time scheduling` | User lacks `rtprio` limits | Same as above |
| `XRUN callback` | Buffer underrun/overrun | Increase buffer size, check CPU governor |
| `Cannot connect to server socket` | JACK server not running | `sudo systemctl start jackd` |
| `ALSA: cannot open audio device hw:USB` | Device busy or wrong name | Check `aplay -l` for correct device name |
| `Cannot create new client` | Too many JACK clients | Close unused clients, increase client limit |
| `Cannot allocate memory` | System out of memory | Reduce buffers, close other applications |
| `subgraph starting at ... timed out` | A JACK client is unresponsive | Restart the stuck client |