From 6a6f2739be0f6a6cfc784023617114050bf8b1fc Mon Sep 17 00:00:00 2001 From: Shawn Date: Tue, 19 May 2026 23:03:19 -0400 Subject: [PATCH] P7-R3: Add comprehensive troubleshooting guide - docs/troubleshooting.md: 8-section guide covering audio, MIDI, network, recording, streaming, system, and build issues - Includes diagnostic health check script and JACK error code reference - All other docs (README, build-guide, user-manual, hardware-compatibility, developer-guide) were already committed and verified as complete --- docs/troubleshooting.md | 714 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 714 insertions(+) create mode 100644 docs/troubleshooting.md diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md new file mode 100644 index 0000000..f78f921 --- /dev/null +++ b/docs/troubleshooting.md @@ -0,0 +1,714 @@ +# Troubleshooting Guide + +Common issues and solutions for the Raspberry Pi Real-Time Audio Mixer. + +## Table of Contents + +1. [Audio Issues](#1-audio-issues) +2. [MIDI Issues](#2-midi-issues) +3. [Network & Web UI Issues](#3-network--web-ui-issues) +4. [Recording Issues](#4-recording-issues) +5. [Streaming Issues](#5-streaming-issues) +6. [System Issues](#6-system-issues) +7. [Build Issues](#7-build-issues) +8. [Diagnostic Tools](#8-diagnostic-tools) + +--- + +## 1. Audio Issues + +### No audio output + +**Symptoms:** Everything looks normal but no sound comes out. + +**Checklist:** +```bash +# 1. Is the USB audio interface detected? +aplay -l +arecord -l +cat /proc/asound/cards + +# 2. Is JACK running? +systemctl status jackd +ps aux | grep jackd + +# 3. Are ports connected? +jack_lsp -c + +# 4. Is the channel unmuted and volume up? +curl -s http://localhost:8080/channels/1 \ + -H "X-API-Key: your-key" | jq '.mute, .volume' + +# 5. Are system outputs unmuted? +alsamixer # Press F6 to select USB device, check levels +``` + +**Common fixes:** +- `sudo systemctl restart jackd && sleep 2 && sudo systemctl restart mixer-api` +- Check `alsamixer` — some interfaces have hardware mute switches +- Verify cables: input on capture_1 not playback_1 +- If using headphones, check the headphone output routing + +--- + +### xruns (audio dropouts, clicks, pops) + +**Symptoms:** Audio glitches, crackling, periodic dropouts, JACK log shows `XRUN`. + +**Check JACK status:** +```bash +# View JACK logs for xrun count +journalctl -u jackd -n 50 | grep -i xrun + +# Check CPU isolation +cat /sys/devices/system/cpu/isolated # should show 1-3 +cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor # should be 'performance' +``` + +**Troubleshooting order:** + +1. **Increase buffer size.** Edit `/etc/systemd/system/jackd.service`: + ``` + -p 128 → -p 256 (or -p 512 for extreme cases) + ``` + Then: `sudo systemctl daemon-reload && sudo systemctl restart jackd` + +2. **Check CPU governor:** + ```bash + cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor + # All should say 'performance'. If not: + sudo systemctl restart cpu-performance + ``` + +3. **Verify RT priorities:** + ```bash + ps -eo pid,cls,rtprio,comm | grep -E 'jackd|irq' + # jackd should have CLS=FF (SCHED_FIFO), RTPRIO=70 + # xhci IRQ thread should have RTPRIO=95 + ``` + +4. **Check for USB bandwidth issues:** + ```bash + lsusb -t # Check which devices share the USB bus + # Move non-audio USB devices to a different hub/port + ``` + +5. **Reduce channel count or effects:** + - Fewer active channels → less DSP load + - Disable unused plugins + - Reduce EQ bands (or disable EQ on unused channels) + +6. **Check real-time group membership:** + ```bash + groups | grep audio # must include 'audio' + ulimit -r # should show 99 + ulimit -l # should show unlimited + ``` + +--- + +### Audio interface not detected + +**Symptoms:** `aplay -l` doesn't show the USB interface. + +**Checklist:** +```bash +# Is the device visible to USB? +lsusb +# Look for your audio interface in the list + +# Kernel messages about USB audio +dmesg | grep -i 'usb\|audio\|snd' | tail -30 + +# Are ALSA modules loaded? +lsmod | grep snd_usb +``` + +**Fixes:** +- Try a different USB port (USB 3.0 blue ports preferred) +- Try a different USB cable (short, shielded cables work best) +- Power-cycle the audio interface +- If bus-powered, use a powered USB hub +- Check the interface is in class-compliant mode (not proprietary driver mode): + - Focusrite: hold 48V button while connecting USB (Gen 3 only) + - Behringer: generally always class-compliant + +--- + +### USB audio interface randomly disconnects + +**Symptoms:** Interface disappears from `aplay -l` mid-session, JACK crashes. + +**Causes and fixes:** + +1. **Under-voltage** — check for lightning bolt on HDMI display or run: + ```bash + vcgencmd get_throttled + # 0x0 = no issues. 0x50000 = under-voltage has occurred + ``` + Fix: Use official RPi PSU (5.1V/3A), shorter/better USB-C cable. + +2. **USB cable** — long or poor-quality USB cables cause disconnects. + Fix: Use a cable ≤2m, USB-IF certified. + +3. **Power saving** — USB autosuspend may disconnect the interface: + ```bash + # Check current setting + cat /sys/module/usbcore/parameters/autosuspend + # Disable autosuspend (-1 = never) + echo -1 | sudo tee /sys/module/usbcore/parameters/autosuspend + ``` + +4. **EMI interference** — nearby power supplies, motors, or RF sources. + Fix: Use ferrite chokes on USB cables, relocate away from interference. + +--- + +### High latency (round-trip >20ms) + +```bash +# Measure round-trip latency +jack_delay + +# Check current buffer settings +jack_bufsize +``` + +**To reduce latency:** +1. Reduce buffer size in `/etc/systemd/system/jackd.service`: + `-p 128` → `-p 64` (more aggressive, may cause xruns) +2. Reduce periods: `-n 3` → `-n 2` +3. Ensure PREEMPT_RT kernel is loaded: + ```bash + uname -r # should contain 'rt' or 'PREEMPT_RT' + ``` +4. Verify kernel cmdline tuning took effect: + ```bash + cat /proc/cmdline # should show isolcpus, threadirqs, etc. + ``` + +--- + +## 2. MIDI Issues + +### MIDI controller not detected + +```bash +# List MIDI devices +aconnect -l +amidi -l + +# Check USB detection +lsusb | grep -i midi + +# Check kernel MIDI driver +dmesg | grep -i midi +``` + +**Fixes:** +- Try a different USB port +- Some controllers need to be in a specific mode (e.g., Akai MIDImix: + hold "SEND ALL" while connecting) +- Ensure `snd-seq` and `snd-usb-audio` modules are loaded: + ```bash + sudo modprobe snd-seq + sudo modprobe snd-usb-audio + ``` + +--- + +### MIDI controller not controlling parameters + +```bash +# Monitor raw MIDI events +aseqdump -p 20:0 # replace with correct port number from aconnect -l + +# Check if MIDI engine is running +systemctl status mixer-api +journalctl -u mixer-api | grep -i midi + +# Check if mappings exist +curl http://localhost:8080/midi/mappings \ + -H "X-API-Key: your-key" +``` + +**Fixes:** +- Use **MIDI Learn** to map controls: + ```bash + curl -X POST http://pi-mixer.local:8080/midi/learn/start \ + -H "X-API-Key: your-key" + # Click parameter in web UI, then move physical control + curl -X POST http://pi-mixer.local:8080/midi/learn/stop \ + -H "X-API-Key: your-key" + ``` +- Check the controller is sending CC messages (not SysEx or NRPN only) +- Verify the MIDI channel matches your mapping + +--- + +### MIDI Clock not syncing + +```bash +# Check clock mode +curl http://localhost:8080/midi/clock \ + -H "X-API-Key: your-key" + +# Set clock mode explicitly +curl -X PUT http://localhost:8080/midi/clock/mode \ + -H "X-API-Key: your-key" \ + -H "Content-Type: application/json" \ + -d '{"mode": "slave"}' +``` + +--- + +## 3. Network & Web UI Issues + +### Web UI not accessible + +**Checklist:** +```bash +# Is the mixer API running? +systemctl status mixer-api +curl http://localhost:8080/stats -H "X-API-Key: your-key" + +# What's the Pi's IP? +ip addr show | grep "inet " + +# Is the port open? +ss -tlnp | grep 8080 + +# Can you reach it from another machine? +ping pi-mixer.local +curl http://pi-mixer.local:8080/stats -H "X-API-Key: your-key" +``` + +**Fixes:** +- Restart the API: `sudo systemctl restart mixer-api` +- Check the service logs: `journalctl -u mixer-api -n 50` +- If `.local` doesn't resolve, use the IP address directly +- Check firewall: `sudo iptables -L` (port 8080 must be open) +- For WiFi, ensure the Pi and your device are on the same network + +--- + +### WebSocket disconnects frequently + +**Symptoms:** Web UI shows "Disconnected" banner, reconnects intermittently. + +**Fixes:** +- Use Ethernet instead of WiFi +- Reduce browser tab count (WebSocket connections consume memory) +- Check API server logs: `journalctl -u mixer-api -f` +- Increase WebSocket ping interval in `src/network/websocket.py` + +--- + +### API returns "Unauthorized" + +**Check API key:** +```bash +# Find your API key +grep API_KEY /etc/systemd/system/mixer-api.service + +# Test with the key +curl http://localhost:8080/stats -H "X-API-Key: your-api-key" +``` + +**To reset the API key:** +```bash +# Generate a new key +NEW_KEY=$(openssl rand -hex 16) +echo "Generated key: $NEW_KEY" + +# Update the service file +sudo sed -i "s/Environment=MIXER_API_KEY=.*/Environment=MIXER_API_KEY=$NEW_KEY/" \ + /etc/systemd/system/mixer-api.service + +# Reload and restart +sudo systemctl daemon-reload +sudo systemctl restart mixer-api +``` + +--- + +### Cannot connect via SSH + +```bash +# Check SSH service +sudo systemctl status ssh + +# Enable if disabled +sudo systemctl enable --now ssh + +# Find the IP +ip addr show | grep "inet " | grep -v 127.0.0.1 +``` + +Default SSH credentials: `pi` / `raspberry`. Change immediately: +```bash +passwd +``` + +--- + +## 4. Recording Issues + +### Recording fails to start + +**Checklist:** +```bash +# Is the data partition mounted? +df -h /data + +# Is there enough space? +df -h /data | awk 'NR==2 {print $4}' # available space + +# Are channels armed? +curl http://localhost:8080/recording/status \ + -H "X-API-Key: your-key" +``` + +**Fixes:** +- Arm at least one channel before recording +- Ensure `/data` partition exists and is writable: + ```bash + sudo mkdir -p /data/recordings + sudo chown pi:pi /data/recordings + ``` +- Free up space: delete old recordings in `/data/recordings/` + +--- + +### Recorded files are corrupt or silent + +**Check:** +```bash +# Check WAV file validity +ffprobe /data/recordings/session_001/channel_01.wav + +# Check file size (non-zero?) +ls -lh /data/recordings/session_001/ +``` + +**Causes:** +- JACK wasn't running or crashed during recording → restart JACK and re-record +- SD card too slow → use A2-class card +- Disk full mid-recording → check disk space before long sessions +- Channel not armed → arm channel before pressing Record + +--- + +### SD card too slow for multi-track recording + +```bash +# Test write speed +dd if=/dev/zero of=/data/test_write bs=1M count=256 conv=fdatasync 2>&1 | \ + tail -1 | awk '{print $8 " " $9}' +# Should show >20 MB/s for 16-track recording + +# Check SD card class +sudo cat /sys/block/mmcblk0/device/cid +# Use the CID to look up the card specs +``` + +**Fixes:** +- Use A2-class SD card (SanDisk Extreme Pro or Samsung EVO Select) +- Reduce track count (16→8 channels) +- Reduce bit depth (32-bit float→24-bit or 16-bit) +- Use external USB SSD for recording: + ```bash + # Mount USB drive and configure recording path + sudo mount /dev/sda1 /mnt/recordings + curl -X PUT http://localhost:8080/recording/path \ + -H "X-API-Key: your-key" \ + -d '{"path": "/mnt/recordings"}' + ``` + +--- + +## 5. Streaming Issues + +### Stream won't start + +**Checklist:** +```bash +# Is GStreamer installed? +which gst-launch-1.0 +gst-launch-1.0 --version + +# Is a camera connected? +v4l2-ctl --list-devices + +# Can you reach the streaming server? +curl -s --max-time 5 rtmp://a.rtmp.youtube.com/live2/ 2>&1 +# Should return RTMP handshake, not timeout +``` + +**Fixes:** +- Install GStreamer: `sudo apt install gstreamer1.0-tools gstreamer1.0-plugins-*` +- Verify stream key is correct (no extra spaces) +- Check internet connection: `ping 8.8.8.8` +- Some networks block RTMP — try on a different network +- Reduce bitrate: 4500 → 2500 Kbps for slow connections + +--- + +### Stream is choppy or dropping frames + +```bash +# Monitor stream stats +curl http://localhost:8080/stream/stats \ + -H "X-API-Key: your-key" +``` + +**Fixes:** +- **Use Ethernet** — WiFi is unreliable for streaming +- Reduce video bitrate (4500→2500 Kbps) and resolution (1080p→720p) +- Reduce audio channels in the stream mix +- Close other network-intensive applications +- Check CPU usage: `top` — streaming should be <50% CPU +- Use hardware-accelerated encoding if available (omx264 instead of x264) + +--- + +### Camera not detected + +```bash +# List video devices +v4l2-ctl --list-devices +ls /dev/video* + +# Check Pi Camera +vcgencmd get_camera +``` + +**Fixes:** +- Pi Camera: ensure ribbon cable is properly seated +- USB Camera: try a different USB port +- Check camera is supported: `v4l2-ctl --list-formats-ext` +- Enable Pi Camera in `raspi-config` → Interface Options → Camera + +--- + +## 6. System Issues + +### Pi won't boot + +**Checklist:** +1. **Power LED** — solid red means power OK. Flashing red = under-voltage. +2. **ACT LED** — green activity light should flash. No flashing = SD card issue. +3. **HDMI display** — connect a monitor to see boot messages. + +**Fixes:** +- Re-flash the SD card +- Try a different power supply (5V/3A) +- Check the SD card is properly inserted +- Verify the image was written correctly: + ```bash + # Compare checksums + sha256sum build/out/rpi-audio-mixer-*.img.xz + ``` + +--- + +### Pi is slow or unresponsive + +```bash +# Check CPU temperature +vcgencmd measure_temp + +# Check CPU frequency +vcgencmd measure_clock arm + +# Check throttling status +vcgencmd get_throttled +# 0x0 = OK +# 0x50000 = under-voltage +# 0x50005 = throttled (temperature >80°C) +``` + +**Fixes:** +- Add a heatsink to the CPU +- Ensure proper ventilation +- Reduce channel count or effects +- Check for background processes: `top` + +--- + +### First-boot wizard keeps appearing + +```bash +# Check if re-arm flag exists +ls -la /force-firstboot + +# Remove it +sudo rm /force-firstboot + +# Disable the first-boot service +sudo systemctl disable mixer-firstboot +``` + +--- + +### Systemd services fail to start + +```bash +# Check overall service status +systemctl --failed + +# Check specific service +journalctl -u mixer-api -n 100 + +# Check dependencies +systemctl list-dependencies mixer-api + +# Verify service files exist +ls -la /etc/systemd/system/mixer-*.service +``` + +**Common fixes:** +- JACK must start before `mixer-api`: + ```bash + sudo systemctl start jackd + sleep 2 + sudo systemctl start mixer-api + ``` +- Permissions issue: ensure `pi` user is in the `audio` group +- Config file syntax error: `systemd-analyze verify /etc/systemd/system/mixer-api.service` + +--- + +### Disk full + +```bash +# Check disk usage +df -h + +# Find large directories +du -sh /data/* | sort -rh | head -10 + +# Clean up +sudo journalctl --vacuum-size=100M # limit journal size +rm -rf /data/recordings/old_session/ # delete old recordings +``` + +--- + +## 7. Build Issues + +### Build fails — see build/README.md for common issues + +Common build problems: + +| Problem | Solution | +|---------|----------| +| `qemu-aarch64-static not found` | `sudo apt install qemu-user-static && sudo update-binfmts --enable qemu-aarch64` | +| `apt fails in chroot` | DNS not resolving. Check host `/etc/resolv.conf`. | +| `No space left on device` | Increase image size: `--image-size 16384` | +| `kernel build fails` | Install deps: `sudo apt install gcc-aarch64-linux-gnu flex bison libssl-dev bc` | +| `loop device busy` | `sudo losetup -D` to detach stale loop devices | +| `permission denied` | Build requires sudo. Ensure passwordless sudo is configured. | + +--- + +## 8. Diagnostic Tools + +### Quick Health Check + +```bash +#!/bin/bash +# mixer-health-check.sh — run on the Pi for a quick diagnostic + +echo "=== Mixer Health Check ===" +echo + +echo "--- Kernel ---" +uname -r | grep -q rt && echo "✅ PREEMPT_RT kernel" || echo "❌ Stock kernel (no RT)" +echo + +echo "--- Audio ---" +aplay -l 2>/dev/null | grep -q USB && echo "✅ USB audio interface detected" || echo "❌ No USB audio" +echo + +echo "--- Services ---" +for svc in jackd mixer-api mixer-ui; do + systemctl is-active --quiet $svc && echo "✅ $svc running" || echo "❌ $svc not running" +done +echo + +echo "--- CPU ---" +cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 2>/dev/null +echo "CPU temp: $(vcgencmd measure_temp)" +echo + +echo "--- Disk ---" +df -h / /data 2>/dev/null +echo + +echo "--- Network ---" +ip -4 addr show | grep "inet " | grep -v 127.0.0.1 +echo + +echo "--- Recent Errors ---" +journalctl -p err -n 5 --no-pager +``` + +### Useful Diagnostic Commands + +```bash +# Audio +aplay -l # List playback devices +arecord -l # List capture devices +jack_lsp -c # JACK ports and connections +jack_bufsize # Current JACK buffer size +cat /proc/asound/cards # Kernel audio device list +amidi -l # List MIDI devices + +# System +vcgencmd get_throttled # Throttling/voltage status +vcgencmd measure_temp # CPU temperature +cat /proc/cmdline # Kernel boot parameters +cat /sys/devices/system/cpu/isolated # CPU isolation +ulimit -r # Max real-time priority +groups # User group membership + +# USB +lsusb -t # USB device tree (bandwidth) +lsusb -v 2>/dev/null | grep -i audio # Audio device details + +# Services +systemctl --failed # Failed services +journalctl -u mixer-api -n 50 # API server logs +journalctl -u jackd -n 50 # JACK logs + +# Network +ip addr show # IP addresses +ss -tlnp # Listening ports +curl localhost:8080/stats # API health check +``` + +### Getting Help + +1. Run the health check script above +2. Collect logs: + ```bash + journalctl -u jackd -u mixer-api -u mixer-ui --since "10 minutes ago" > mixer-logs.txt + ``` +3. Include `dmesg | tail -50` output +4. Report hardware details: `cat /proc/cpuinfo | grep Model`, `lsusb` + +--- + +## Appendix: JACK Error Messages Decoded + +| Error | Meaning | Fix | +|-------|---------|-----| +| `Cannot lock down memory` | User lacks `memlock` limits | Add user to `audio` group, check `/etc/security/limits.d/99-audio.conf` | +| `Cannot use real-time scheduling` | User lacks `rtprio` limits | Same as above | +| `XRUN callback` | Buffer underrun/overrun | Increase buffer size, check CPU governor | +| `Cannot connect to server socket` | JACK server not running | `sudo systemctl start jackd` | +| `ALSA: cannot open audio device hw:USB` | Device busy or wrong name | Check `aplay -l` for correct device name | +| `Cannot create new client` | Too many JACK clients | Close unused clients, increase client limit | +| `Cannot allocate memory` | System out of memory | Reduce buffers, close other applications | +| `subgraph starting at ... timed out` | A JACK client is unresponsive | Restart the stuck client |