P7-R3: Add comprehensive troubleshooting guide

- docs/troubleshooting.md: 8-section guide covering audio, MIDI, network, recording, streaming, system, and build issues - Includes diagnostic health check script and JACK error code reference - All other docs (README, build-guide, user-manual, hardware-compatibility, developer-guide) were already committed and verified as complete
2026-05-19 23:03:19 -04:00
parent e450189858
commit 6a6f2739be
1 changed files with 714 additions and 0 deletions
@@ -0,0 +1,714 @@
+# Troubleshooting Guide
+
+Common issues and solutions for the Raspberry Pi Real-Time Audio Mixer.
+
+## Table of Contents
+
+1. [Audio Issues](#1-audio-issues)
+2. [MIDI Issues](#2-midi-issues)
+3. [Network & Web UI Issues](#3-network--web-ui-issues)
+4. [Recording Issues](#4-recording-issues)
+5. [Streaming Issues](#5-streaming-issues)
+6. [System Issues](#6-system-issues)
+7. [Build Issues](#7-build-issues)
+8. [Diagnostic Tools](#8-diagnostic-tools)
+
+---
+
+## 1. Audio Issues
+
+### No audio output
+
+**Symptoms:** Everything looks normal but no sound comes out.
+
+**Checklist:**
+```bash
+# 1. Is the USB audio interface detected?
+aplay -l
+arecord -l
+cat /proc/asound/cards
+
+# 2. Is JACK running?
+systemctl status jackd
+ps aux | grep jackd
+
+# 3. Are ports connected?
+jack_lsp -c
+
+# 4. Is the channel unmuted and volume up?
+curl -s http://localhost:8080/channels/1 \
+     -H "X-API-Key: your-key" | jq '.mute, .volume'
+
+# 5. Are system outputs unmuted?
+alsamixer    # Press F6 to select USB device, check levels
+```
+
+**Common fixes:**
+- `sudo systemctl restart jackd && sleep 2 && sudo systemctl restart mixer-api`
+- Check `alsamixer` — some interfaces have hardware mute switches
+- Verify cables: input on capture_1 not playback_1
+- If using headphones, check the headphone output routing
+
+---
+
+### xruns (audio dropouts, clicks, pops)
+
+**Symptoms:** Audio glitches, crackling, periodic dropouts, JACK log shows `XRUN`.
+
+**Check JACK status:**
+```bash
+# View JACK logs for xrun count
+journalctl -u jackd -n 50 | grep -i xrun
+
+# Check CPU isolation
+cat /sys/devices/system/cpu/isolated            # should show 1-3
+cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor  # should be 'performance'
+```
+
+**Troubleshooting order:**
+
+1. **Increase buffer size.** Edit `/etc/systemd/system/jackd.service`:
+   ```
+   -p 128 → -p 256 (or -p 512 for extreme cases)
+   ```
+   Then: `sudo systemctl daemon-reload && sudo systemctl restart jackd`
+
+2. **Check CPU governor:**
+   ```bash
+   cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
+   # All should say 'performance'. If not:
+   sudo systemctl restart cpu-performance
+   ```
+
+3. **Verify RT priorities:**
+   ```bash
+   ps -eo pid,cls,rtprio,comm | grep -E 'jackd|irq'
+   # jackd should have CLS=FF (SCHED_FIFO), RTPRIO=70
+   # xhci IRQ thread should have RTPRIO=95
+   ```
+
+4. **Check for USB bandwidth issues:**
+   ```bash
+   lsusb -t    # Check which devices share the USB bus
+   # Move non-audio USB devices to a different hub/port
+   ```
+
+5. **Reduce channel count or effects:**
+   - Fewer active channels → less DSP load
+   - Disable unused plugins
+   - Reduce EQ bands (or disable EQ on unused channels)
+
+6. **Check real-time group membership:**
+   ```bash
+   groups | grep audio     # must include 'audio'
+   ulimit -r               # should show 99
+   ulimit -l               # should show unlimited
+   ```
+
+---
+
+### Audio interface not detected
+
+**Symptoms:** `aplay -l` doesn't show the USB interface.
+
+**Checklist:**
+```bash
+# Is the device visible to USB?
+lsusb
+# Look for your audio interface in the list
+
+# Kernel messages about USB audio
+dmesg | grep -i 'usb\|audio\|snd' | tail -30
+
+# Are ALSA modules loaded?
+lsmod | grep snd_usb
+```
+
+**Fixes:**
+- Try a different USB port (USB 3.0 blue ports preferred)
+- Try a different USB cable (short, shielded cables work best)
+- Power-cycle the audio interface
+- If bus-powered, use a powered USB hub
+- Check the interface is in class-compliant mode (not proprietary driver mode):
+  - Focusrite: hold 48V button while connecting USB (Gen 3 only)
+  - Behringer: generally always class-compliant
+
+---
+
+### USB audio interface randomly disconnects
+
+**Symptoms:** Interface disappears from `aplay -l` mid-session, JACK crashes.
+
+**Causes and fixes:**
+
+1. **Under-voltage** — check for lightning bolt on HDMI display or run:
+   ```bash
+   vcgencmd get_throttled
+   # 0x0 = no issues. 0x50000 = under-voltage has occurred
+   ```
+   Fix: Use official RPi PSU (5.1V/3A), shorter/better USB-C cable.
+
+2. **USB cable** — long or poor-quality USB cables cause disconnects.
+   Fix: Use a cable ≤2m, USB-IF certified.
+
+3. **Power saving** — USB autosuspend may disconnect the interface:
+   ```bash
+   # Check current setting
+   cat /sys/module/usbcore/parameters/autosuspend
+   # Disable autosuspend (-1 = never)
+   echo -1 | sudo tee /sys/module/usbcore/parameters/autosuspend
+   ```
+
+4. **EMI interference** — nearby power supplies, motors, or RF sources.
+   Fix: Use ferrite chokes on USB cables, relocate away from interference.
+
+---
+
+### High latency (round-trip >20ms)
+
+```bash
+# Measure round-trip latency
+jack_delay
+
+# Check current buffer settings
+jack_bufsize
+```
+
+**To reduce latency:**
+1. Reduce buffer size in `/etc/systemd/system/jackd.service`:
+   `-p 128` → `-p 64` (more aggressive, may cause xruns)
+2. Reduce periods: `-n 3` → `-n 2`
+3. Ensure PREEMPT_RT kernel is loaded:
+   ```bash
+   uname -r   # should contain 'rt' or 'PREEMPT_RT'
+   ```
+4. Verify kernel cmdline tuning took effect:
+   ```bash
+   cat /proc/cmdline   # should show isolcpus, threadirqs, etc.
+   ```
+
+---
+
+## 2. MIDI Issues
+
+### MIDI controller not detected
+
+```bash
+# List MIDI devices
+aconnect -l
+amidi -l
+
+# Check USB detection
+lsusb | grep -i midi
+
+# Check kernel MIDI driver
+dmesg | grep -i midi
+```
+
+**Fixes:**
+- Try a different USB port
+- Some controllers need to be in a specific mode (e.g., Akai MIDImix:
+  hold "SEND ALL" while connecting)
+- Ensure `snd-seq` and `snd-usb-audio` modules are loaded:
+  ```bash
+  sudo modprobe snd-seq
+  sudo modprobe snd-usb-audio
+  ```
+
+---
+
+### MIDI controller not controlling parameters
+
+```bash
+# Monitor raw MIDI events
+aseqdump -p 20:0   # replace with correct port number from aconnect -l
+
+# Check if MIDI engine is running
+systemctl status mixer-api
+journalctl -u mixer-api | grep -i midi
+
+# Check if mappings exist
+curl http://localhost:8080/midi/mappings \
+     -H "X-API-Key: your-key"
+```
+
+**Fixes:**
+- Use **MIDI Learn** to map controls:
+  ```bash
+  curl -X POST http://pi-mixer.local:8080/midi/learn/start \
+       -H "X-API-Key: your-key"
+  # Click parameter in web UI, then move physical control
+  curl -X POST http://pi-mixer.local:8080/midi/learn/stop \
+       -H "X-API-Key: your-key"
+  ```
+- Check the controller is sending CC messages (not SysEx or NRPN only)
+- Verify the MIDI channel matches your mapping
+
+---
+
+### MIDI Clock not syncing
+
+```bash
+# Check clock mode
+curl http://localhost:8080/midi/clock \
+     -H "X-API-Key: your-key"
+
+# Set clock mode explicitly
+curl -X PUT http://localhost:8080/midi/clock/mode \
+     -H "X-API-Key: your-key" \
+     -H "Content-Type: application/json" \
+     -d '{"mode": "slave"}'
+```
+
+---
+
+## 3. Network & Web UI Issues
+
+### Web UI not accessible
+
+**Checklist:**
+```bash
+# Is the mixer API running?
+systemctl status mixer-api
+curl http://localhost:8080/stats -H "X-API-Key: your-key"
+
+# What's the Pi's IP?
+ip addr show | grep "inet "
+
+# Is the port open?
+ss -tlnp | grep 8080
+
+# Can you reach it from another machine?
+ping pi-mixer.local
+curl http://pi-mixer.local:8080/stats -H "X-API-Key: your-key"
+```
+
+**Fixes:**
+- Restart the API: `sudo systemctl restart mixer-api`
+- Check the service logs: `journalctl -u mixer-api -n 50`
+- If `.local` doesn't resolve, use the IP address directly
+- Check firewall: `sudo iptables -L` (port 8080 must be open)
+- For WiFi, ensure the Pi and your device are on the same network
+
+---
+
+### WebSocket disconnects frequently
+
+**Symptoms:** Web UI shows "Disconnected" banner, reconnects intermittently.
+
+**Fixes:**
+- Use Ethernet instead of WiFi
+- Reduce browser tab count (WebSocket connections consume memory)
+- Check API server logs: `journalctl -u mixer-api -f`
+- Increase WebSocket ping interval in `src/network/websocket.py`
+
+---
+
+### API returns "Unauthorized"
+
+**Check API key:**
+```bash
+# Find your API key
+grep API_KEY /etc/systemd/system/mixer-api.service
+
+# Test with the key
+curl http://localhost:8080/stats -H "X-API-Key: your-api-key"
+```
+
+**To reset the API key:**
+```bash
+# Generate a new key
+NEW_KEY=$(openssl rand -hex 16)
+echo "Generated key: $NEW_KEY"
+
+# Update the service file
+sudo sed -i "s/Environment=MIXER_API_KEY=.*/Environment=MIXER_API_KEY=$NEW_KEY/" \
+    /etc/systemd/system/mixer-api.service
+
+# Reload and restart
+sudo systemctl daemon-reload
+sudo systemctl restart mixer-api
+```
+
+---
+
+### Cannot connect via SSH
+
+```bash
+# Check SSH service
+sudo systemctl status ssh
+
+# Enable if disabled
+sudo systemctl enable --now ssh
+
+# Find the IP
+ip addr show | grep "inet " | grep -v 127.0.0.1
+```
+
+Default SSH credentials: `pi` / `raspberry`. Change immediately:
+```bash
+passwd
+```
+
+---
+
+## 4. Recording Issues
+
+### Recording fails to start
+
+**Checklist:**
+```bash
+# Is the data partition mounted?
+df -h /data
+
+# Is there enough space?
+df -h /data | awk 'NR==2 {print $4}'  # available space
+
+# Are channels armed?
+curl http://localhost:8080/recording/status \
+     -H "X-API-Key: your-key"
+```
+
+**Fixes:**
+- Arm at least one channel before recording
+- Ensure `/data` partition exists and is writable:
+  ```bash
+  sudo mkdir -p /data/recordings
+  sudo chown pi:pi /data/recordings
+  ```
+- Free up space: delete old recordings in `/data/recordings/`
+
+---
+
+### Recorded files are corrupt or silent
+
+**Check:**
+```bash
+# Check WAV file validity
+ffprobe /data/recordings/session_001/channel_01.wav
+
+# Check file size (non-zero?)
+ls -lh /data/recordings/session_001/
+```
+
+**Causes:**
+- JACK wasn't running or crashed during recording → restart JACK and re-record
+- SD card too slow → use A2-class card
+- Disk full mid-recording → check disk space before long sessions
+- Channel not armed → arm channel before pressing Record
+
+---
+
+### SD card too slow for multi-track recording
+
+```bash
+# Test write speed
+dd if=/dev/zero of=/data/test_write bs=1M count=256 conv=fdatasync 2>&1 | \
+    tail -1 | awk '{print $8 " " $9}'
+# Should show >20 MB/s for 16-track recording
+
+# Check SD card class
+sudo cat /sys/block/mmcblk0/device/cid
+# Use the CID to look up the card specs
+```
+
+**Fixes:**
+- Use A2-class SD card (SanDisk Extreme Pro or Samsung EVO Select)
+- Reduce track count (16→8 channels)
+- Reduce bit depth (32-bit float→24-bit or 16-bit)
+- Use external USB SSD for recording:
+  ```bash
+  # Mount USB drive and configure recording path
+  sudo mount /dev/sda1 /mnt/recordings
+  curl -X PUT http://localhost:8080/recording/path \
+       -H "X-API-Key: your-key" \
+       -d '{"path": "/mnt/recordings"}'
+  ```
+
+---
+
+## 5. Streaming Issues
+
+### Stream won't start
+
+**Checklist:**
+```bash
+# Is GStreamer installed?
+which gst-launch-1.0
+gst-launch-1.0 --version
+
+# Is a camera connected?
+v4l2-ctl --list-devices
+
+# Can you reach the streaming server?
+curl -s --max-time 5 rtmp://a.rtmp.youtube.com/live2/ 2>&1
+# Should return RTMP handshake, not timeout
+```
+
+**Fixes:**
+- Install GStreamer: `sudo apt install gstreamer1.0-tools gstreamer1.0-plugins-*`
+- Verify stream key is correct (no extra spaces)
+- Check internet connection: `ping 8.8.8.8`
+- Some networks block RTMP — try on a different network
+- Reduce bitrate: 4500 → 2500 Kbps for slow connections
+
+---
+
+### Stream is choppy or dropping frames
+
+```bash
+# Monitor stream stats
+curl http://localhost:8080/stream/stats \
+     -H "X-API-Key: your-key"
+```
+
+**Fixes:**
+- **Use Ethernet** — WiFi is unreliable for streaming
+- Reduce video bitrate (4500→2500 Kbps) and resolution (1080p→720p)
+- Reduce audio channels in the stream mix
+- Close other network-intensive applications
+- Check CPU usage: `top` — streaming should be <50% CPU
+- Use hardware-accelerated encoding if available (omx264 instead of x264)
+
+---
+
+### Camera not detected
+
+```bash
+# List video devices
+v4l2-ctl --list-devices
+ls /dev/video*
+
+# Check Pi Camera
+vcgencmd get_camera
+```
+
+**Fixes:**
+- Pi Camera: ensure ribbon cable is properly seated
+- USB Camera: try a different USB port
+- Check camera is supported: `v4l2-ctl --list-formats-ext`
+- Enable Pi Camera in `raspi-config` → Interface Options → Camera
+
+---
+
+## 6. System Issues
+
+### Pi won't boot
+
+**Checklist:**
+1. **Power LED** — solid red means power OK. Flashing red = under-voltage.
+2. **ACT LED** — green activity light should flash. No flashing = SD card issue.
+3. **HDMI display** — connect a monitor to see boot messages.
+
+**Fixes:**
+- Re-flash the SD card
+- Try a different power supply (5V/3A)
+- Check the SD card is properly inserted
+- Verify the image was written correctly:
+  ```bash
+  # Compare checksums
+  sha256sum build/out/rpi-audio-mixer-*.img.xz
+  ```
+
+---
+
+### Pi is slow or unresponsive
+
+```bash
+# Check CPU temperature
+vcgencmd measure_temp
+
+# Check CPU frequency
+vcgencmd measure_clock arm
+
+# Check throttling status
+vcgencmd get_throttled
+# 0x0 = OK
+# 0x50000 = under-voltage
+# 0x50005 = throttled (temperature >80°C)
+```
+
+**Fixes:**
+- Add a heatsink to the CPU
+- Ensure proper ventilation
+- Reduce channel count or effects
+- Check for background processes: `top`
+
+---
+
+### First-boot wizard keeps appearing
+
+```bash
+# Check if re-arm flag exists
+ls -la /force-firstboot
+
+# Remove it
+sudo rm /force-firstboot
+
+# Disable the first-boot service
+sudo systemctl disable mixer-firstboot
+```
+
+---
+
+### Systemd services fail to start
+
+```bash
+# Check overall service status
+systemctl --failed
+
+# Check specific service
+journalctl -u mixer-api -n 100
+
+# Check dependencies
+systemctl list-dependencies mixer-api
+
+# Verify service files exist
+ls -la /etc/systemd/system/mixer-*.service
+```
+
+**Common fixes:**
+- JACK must start before `mixer-api`:
+  ```bash
+  sudo systemctl start jackd
+  sleep 2
+  sudo systemctl start mixer-api
+  ```
+- Permissions issue: ensure `pi` user is in the `audio` group
+- Config file syntax error: `systemd-analyze verify /etc/systemd/system/mixer-api.service`
+
+---
+
+### Disk full
+
+```bash
+# Check disk usage
+df -h
+
+# Find large directories
+du -sh /data/* | sort -rh | head -10
+
+# Clean up
+sudo journalctl --vacuum-size=100M     # limit journal size
+rm -rf /data/recordings/old_session/  # delete old recordings
+```
+
+---
+
+## 7. Build Issues
+
+### Build fails — see build/README.md for common issues
+
+Common build problems:
+
+| Problem | Solution |
+|---------|----------|
+| `qemu-aarch64-static not found` | `sudo apt install qemu-user-static && sudo update-binfmts --enable qemu-aarch64` |
+| `apt fails in chroot` | DNS not resolving. Check host `/etc/resolv.conf`. |
+| `No space left on device` | Increase image size: `--image-size 16384` |
+| `kernel build fails` | Install deps: `sudo apt install gcc-aarch64-linux-gnu flex bison libssl-dev bc` |
+| `loop device busy` | `sudo losetup -D` to detach stale loop devices |
+| `permission denied` | Build requires sudo. Ensure passwordless sudo is configured. |
+
+---
+
+## 8. Diagnostic Tools
+
+### Quick Health Check
+
+```bash
+#!/bin/bash
+# mixer-health-check.sh — run on the Pi for a quick diagnostic
+
+echo "=== Mixer Health Check ==="
+echo
+
+echo "--- Kernel ---"
+uname -r | grep -q rt && echo "✅ PREEMPT_RT kernel" || echo "❌ Stock kernel (no RT)"
+echo
+
+echo "--- Audio ---"
+aplay -l 2>/dev/null | grep -q USB && echo "✅ USB audio interface detected" || echo "❌ No USB audio"
+echo
+
+echo "--- Services ---"
+for svc in jackd mixer-api mixer-ui; do
+    systemctl is-active --quiet $svc && echo "✅ $svc running" || echo "❌ $svc not running"
+done
+echo
+
+echo "--- CPU ---"
+cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null
+echo "CPU temp: $(vcgencmd measure_temp)"
+echo
+
+echo "--- Disk ---"
+df -h / /data 2>/dev/null
+echo
+
+echo "--- Network ---"
+ip -4 addr show | grep "inet " | grep -v 127.0.0.1
+echo
+
+echo "--- Recent Errors ---"
+journalctl -p err -n 5 --no-pager
+```
+
+### Useful Diagnostic Commands
+
+```bash
+# Audio
+aplay -l                                # List playback devices
+arecord -l                              # List capture devices
+jack_lsp -c                             # JACK ports and connections
+jack_bufsize                            # Current JACK buffer size
+cat /proc/asound/cards                  # Kernel audio device list
+amidi -l                                # List MIDI devices
+
+# System
+vcgencmd get_throttled                  # Throttling/voltage status
+vcgencmd measure_temp                   # CPU temperature
+cat /proc/cmdline                       # Kernel boot parameters
+cat /sys/devices/system/cpu/isolated    # CPU isolation
+ulimit -r                               # Max real-time priority
+groups                                  # User group membership
+
+# USB
+lsusb -t                                # USB device tree (bandwidth)
+lsusb -v 2>/dev/null | grep -i audio    # Audio device details
+
+# Services
+systemctl --failed                      # Failed services
+journalctl -u mixer-api -n 50           # API server logs
+journalctl -u jackd -n 50               # JACK logs
+
+# Network
+ip addr show                            # IP addresses
+ss -tlnp                                # Listening ports
+curl localhost:8080/stats               # API health check
+```
+
+### Getting Help
+
+1. Run the health check script above
+2. Collect logs:
+   ```bash
+   journalctl -u jackd -u mixer-api -u mixer-ui --since "10 minutes ago" > mixer-logs.txt
+   ```
+3. Include `dmesg | tail -50` output
+4. Report hardware details: `cat /proc/cpuinfo | grep Model`, `lsusb`
+
+---
+
+## Appendix: JACK Error Messages Decoded
+
+| Error | Meaning | Fix |
+|-------|---------|-----|
+| `Cannot lock down memory` | User lacks `memlock` limits | Add user to `audio` group, check `/etc/security/limits.d/99-audio.conf` |
+| `Cannot use real-time scheduling` | User lacks `rtprio` limits | Same as above |
+| `XRUN callback` | Buffer underrun/overrun | Increase buffer size, check CPU governor |
+| `Cannot connect to server socket` | JACK server not running | `sudo systemctl start jackd` |
+| `ALSA: cannot open audio device hw:USB` | Device busy or wrong name | Check `aplay -l` for correct device name |
+| `Cannot create new client` | Too many JACK clients | Close unused clients, increase client limit |
+| `Cannot allocate memory` | System out of memory | Reduce buffers, close other applications |
+| `subgraph starting at ... timed out` | A JACK client is unresponsive | Restart the stuck client |