# RT Performance Tuning — Pi Multi-FX Pedal

> Reference doc for real-time audio performance on Raspberry Pi 4B.
> Targets: <12ms round-trip latency (ideally <8ms), zero xruns, CPU <40%.

## Overview

The Pi Multi-FX Pedal runs a JACK audio server with an ALSA backend on a
Raspberry Pi 4B. The signal path is:

```
Guitar → Focusrite 2i2 → ALSA → JACK → Python pipeline (NAM + FX) → JACK → ALSA → Output
```

Each stage adds latency. The total round-trip latency is dominated by:

1. **ALSA period size** (`-p`): The buffer size in frames JACK exchanges
   with the audio hardware. *This is the #1 tuning knob.*
2. **Number of periods** (`-n`): ALSA ring buffer depth. More periods =
   more tolerance for scheduling jitter but higher latency.
3. **Sample rate** (`-r`): Higher rate = lower per-frame latency but more
   CPU. 48kHz is the sweet spot for USB audio interfaces.
4. **NAM inference time**: The C++ subprocess takes 2-5ms per block on
   Pi 4B. This is the bottleneck that sets the minimum viable buffer size.

## Recommended Settings (Pi 4B)

| Setting | Standard | Low Latency | Ultra Low | Unit |
|---------|----------|-------------|-----------|------|
| Period (buffer) | 512 | 256 | 128 | frames |
| Sample rate | 48000 | 48000 | 48000 | Hz |
| Periods (nperiods) | 2 | 2 | 3 | |
| RT priority | 70 | 75 | 80 | |
| Expected latency | ~10.7ms | ~5.3ms | ~2.7ms | |
| Expected NAM CPU | 35-50% | 60-93% | 80-100%+ | |
| Xrun stability | ✅ Stable | ⚠️ Possible | ❌ Likely | |

### Default recommendation: 512/48k (standard)

The standard profile (512 frames, 48kHz, 2 periods) is the **recommended
default** for the Pi 4B. This provides:

- **10.67ms** callback window (more than enough for 2-5ms NAM inference)
- **35-50%** CPU load with LSTM NAM models
- **Zero xruns** in normal playing
- Enough headroom for the FX chain (filters, modulation, reverb)

Even at 512 frames, the **round-trip latency** (capture → process →
playback) is typically **6-10ms** with a USB audio interface — well under
the <12ms target. The round-trip includes two ALSA transfers (capture +
playback), which is why it's lower than the raw period calculation.

## Tuning Knobs

### 1. JACK buffer size (`--period` / `-p`)

The JACK period is the number of frames per audio block. Lower = lower
latency but more CPU and more xrun risk.

```bash
# Current: 512 frames at 48kHz = 10.67ms
jackd -p 512 -r 48000 ...

# Aggressive: 128 frames at 48kHz = 2.67ms
jackd -p 128 -r 48000 ...

# Conservative: 1024 frames at 48kHz = 21.33ms (safe, higher latency)
jackd -p 1024 -r 48000 ...
```

**Measurement:** When you change the period in the UI, the server:
1. Updates `AudioConfig.period` 
2. Updates `LATENCY_PROFILES["custom"]`
3. Stops JACK (with bt-a2dp dance)
4. Updates NAM block size (`set_block_size()`)
5. Updates pipeline DSP (`set_audio_profile()`)
6. Restarts JACK with new period
7. Reconnects FX ports
8. Restarts bt-a2dp

**Timeout caveat:** The UI's POST must use `timeout: 15000` (15s) because
JACK restart takes 6-10s on Pi 4B.

### 2. Number of periods (`--nperiods` / `-n`)

The ALSA period count controls the ring buffer depth:

- **nperiods=2** (default): Lower latency, less tolerance for scheduling
  jitter. Good for stable USB audio interfaces.
- **nperiods=3**: More tolerance for scheduling jitter at the cost of
  ~50% more ALSA buffer latency. Recommended when pushing below 256
  frames where every microsecond counts.

The nperiods trade-off: at 128/48k, nperiods=3 adds 128×3/48000 = 8ms
of ALSA buffer vs 128×2/48000 = 5.3ms for nperiods=2. The extra 2.7ms
can prevent xruns when the CPU governor ramps or a system timer fires.

### 3. RT priority (`-P` / `rt_priority`)

JACK uses `-P` to set SCHED_FIFO priority. On RPi 4B:

| Priority | Effect |
|----------|--------|
| 60 | Default — works but shares CPU with other RT tasks |
| 70 | **Standard profile** — good balance |
| 80 | **Low latency profile** — less scheduling jitter |
| 90-95 | Aggressive — use if xruns persist at 256/48k |

The Python pedal process should also run with RT scheduling:
```bash
chrt -f 80 python3 main.py
```

The systemd service (`pi-multifx-pedal.service`) now wraps this
automatically via `ExecStart=/usr/bin/chrt -f 80 python3 main.py`.

**Requirements:**
- `LimitRTPRIO=95` in the systemd unit (already present)
- `@audio - rtprio 95` in `/etc/security/limits.d/99-audio.conf`
- Process must run as root or with `CAP_SYS_NICE`

### 4. CPU governor → performance

The RPi 4B's CPU governor defaults to `ondemand` or `powersave`, which
keeps the CPU at 600MHz idle and ramps up under load. The ramp-up takes
1-2ms — significant at 256/48k (5.33ms callback window).

**Applied in `main.py` at boot:**
```python
for c in range(os.cpu_count() or 1):
    with open(f"/sys/devices/system/cpu/cpu{c}/cpufreq/scaling_governor", "w") as f:
        f.write("performance")
```

**Verify:**
```bash
cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
# All should show "performance"
```

**Trade-off:** ~0.5W extra power draw (CPU stays at 1.5GHz).

### 5. IRQ affinity — pin USB audio to a dedicated core

On RPi 4B, interrupts are distributed across all 4 cores by default.
Pinning the USB audio IRQ to **core 3** isolates it from kernel
housekeeping on cores 0-2.

**Applied in `main.py` at boot:**
```python
# Find xhci-hcd or dwc_otg IRQ → pin to core 3 (mask 0x8)
echo "8" > /proc/irq/<IRQ>/smp_affinity
echo "3" > /proc/irq/<IRQ>/smp_affinity_list
# All other IRQs moved to cores 0-2 (mask 0x7)
```

**Verify:**
```bash
./scripts/rt-tune.sh --status
# or manually:
cat /proc/irq/*/smp_affinity | sort | uniq -c
```

### 6. mlockall() — lock process memory

Prevents page faults in the RT callback by locking all process pages
in RAM.

**Applied in `main.py` at boot:**
```python
import ctypes
libc = ctypes.CDLL('libc.so.6')
libc.mlockall(3)  # MCL_CURRENT | MCL_FUTURE
```

**Requires:** `LimitMEMLOCK=infinity` in the systemd unit (already present).

**Verify:**
```bash
grep -i lock /proc/$(pidof python3)/status | head -5
# VmLck should be non-zero
```

### 7. GC disable — prevent Python GC pauses

Python's default GC (threshold=700) triggers every ~1.4s in the audio
pipeline due to ~500 numpy allocations/second. Each 10-50ms GC pause
causes audible pops.

**Applied in `main.py`:**
```python
import gc
gc.disable()
gc.collect()  # one final sweep
```

Periodic GC on the HTTP thread (never in RT callback):
```python
import gc
gc.collect()  # in get_state() handler, ~2s poll
```

**Trade-off:** Reference counting handles 99% of cleanup. The OS reclaims
all memory on process exit. Safe for a long-running daemon.

## Measurement Tools

### Round-trip latency (jack_iodelay)

Requires a physical loopback cable (output → input).

```bash
# Quick measurement
jack_iodelay

# Automated (8 samples)
python3 -c "
from src.system.audio import AudioSystem
AudioSystem.measure_roundtrip_latency(samples=8)
"
```

**Skip if:** no loopback cable. The UI round-trip latency is
approximately `2 × period / rate × 1000ms` (capture + playback):

| Period | Rate | Calc. RT latency | Real RT latency |
|--------|------|------------------|-----------------|
| 512   | 48k  | 21.33ms | ~6-10ms (USB interface) |
| 256   | 48k  | 10.67ms | ~4-6ms |
| 128   | 48k  | 5.33ms | ~2-4ms |

The real RT latency is lower than the formula because the USB interface
and ALSA driver pipeline the transfers.

### XRun monitoring

```bash
# Enable kernel tracking
echo 3 | sudo tee /proc/asound/card*/xrun_debug

# Quick check
jack_showtime -c | grep xruns

# Automated monitor (5 min)
python3 -c "
from src.system.audio import AudioSystem
asys = AudioSystem()
result = asys.monitor_xruns(duration=300, interval=10)
print(result)
"
```

**XRun debug bits:**
- Bit 0 (1): Log xruns to kernel ring buffer (`dmesg | grep xrun`)
- Bit 1 (2): Show stack backtrace
- Bit 2 (4): Inhibit xruns (test mode — disables recovery)

The pedal enables bit 0+1 (value 3) at boot for diagnostics.

### NAM CPU load

The state API (`GET /api/state`) now includes `nam_cpu` — the percentage
of the callback window spent in NAM inference:

```bash
curl -s http://pedal.local/api/state | python3 -c "
import json,sys
s = json.load(sys.stdin)
print(f'NAM CPU: {s[\"nam_cpu\"]:.1f}%')
print(f'System CPU: {s[\"cpu_percent\"]:.1f}%')
print(f'Input level: {s[\"input_level\"]:.3f}')
"
```

**Expected values (Pi 4B, LSTM NAM model):**

| Buffer | NAM CPU | Notes |
|--------|---------|-------|
| 64     | 180-200% | xruns guaranteed |
| 128    | 80-100%  | xruns likely |
| 256    | 60-93%   | xruns possible |
| 512    | 35-50%   | **stable** |
| 1024   | 15-25%   | safe, higher latency |

## Systematic Tuning Procedure

### Step 1: Establish baseline

With the current settings, run:

```bash
# 1. Check current profile
curl -s http://pedal.local/api/audio/profile | python3 -m json.tool

# 2. Monitor xruns for 5 minutes
timeout 300 bash -c '
  while true; do
    xruns=$(jack_showtime -c 2>/dev/null | grep xruns)
    echo "$(date +%H:%M:%S) $xruns"
    sleep 10
  done
'

# 3. Measure NAM CPU
curl -s http://pedal.local/api/state | python3 -c "
import json,sys
s = json.load(sys.stdin)
print(f'nam_cpu={s[\"nam_cpu\"]}%  sys_cpu={s[\"cpu_percent\"]}%  '
      f'input={s[\"input_level\"]:.3f}  output={s[\"output_level\"]:.3f}')
"
```

### Step 2: Sweep buffer sizes

For each size (512 → 256 → 128 → 64), test for 5 minutes:

```bash
for period in 512 256 128 64; do
  echo "=== Testing period=$period ==="
  curl -s -X POST -d "{\"period\":$period}" http://pedal.local/api/audio/profile
  sleep 15  # wait for JACK restart + stabilization

  # Check NAM CPU
  curl -s http://pedal.local/api/state | python3 -c "
import json,sys; s=json.load(sys.stdin); print(f'  nam_cpu={s[\"nam_cpu\"]}%')"

  # Monitor 5 min
  python3 -c "
from src.system.audio import AudioSystem
r = AudioSystem().monitor_xruns(300, 10)
print(f'  xruns={r[\"xrun_total\"]} rate={r[\"xrun_rate_per_min\"]}/min stable={r[\"stable\"]}')
"
done
```

### Step 3: Evaluate nperiods sweep

For the best buffer candidates, test nperiods=2 vs nperiods=3:

```bash
for period in 128 256; do
  for nperiods in 2 3; do
    echo "=== p=$period n=$nperiods ==="
    # Manually restart JACK with -n $nperiods
    ssh pedal "sudo killall jackd; sleep 1; \
      jackd -P 70 -d alsa -d hw:0,0 -r 48000 -p $period -n $nperiods -i 2 -o 2 &"
    sleep 5
    # Test...
  done
done
```

### Step 4: Select optimal

Choose the lowest period that achieves zero xruns over a 30-minute
test with active playing. Save to config:

```bash
curl -s -X POST -d '{"period":512,"rate":48000}' \
  http://pedal.local/api/audio/profile
```

## Config File Reference

Key fields in `~/.pedal/config.yaml`:

```yaml
audio:
  profile: custom           # or "standard", "low", "stable"
  period: 512               # frames (64-2048, powers of 2)
  rate: 48000              # Hz (44100, 48000, 96000, 192000)
  input_device: hw:0,0     # ALSA device for capture
  output_device: hw:0,0    # ALSA device for playback
  mode: mono               # or "stereo_4cm"
  jack_enabled: true
  auto_connect: true        # re-connect JACK ports on restart
  hat_type: audioinjector   # or "focusrite_2i2_3gen"

notes:
  - RT tuning March 2025: standard=512/48k stable on Pi 4B with LSTM NAM
  - Changing period/rate in UI saves to these fields automatically
  - On restart, period/rate overrides the profile defaults
```

## Common Issues

### Pops/crackle at 256 frames

**Cause:** NAM inference takes 2-5ms on Pi 4B. At 256/48k (5.33ms window),
there's only 0.33-3.33ms headroom. Any scheduling jitter causes xrun.

**Fixes (in order of effectiveness):**
1. Increase to 512 frames (10.67ms window)
2. Set CPU governor to performance (already done at boot)
3. Pin USB audio IRQ to dedicated core (already done at boot)
4. Use nperiods=3 for more ALSA buffer tolerance
5. Disable Wi-Fi/BT if not needed (both share the USB bus on Pi 4B)
6. Use a lighter NAM model (Feather/Nano instead of LSTM)

### Audio drops out after profile change

**Check the startup order:**
1. NAM block size must be set BEFORE `jack_client.start()`
2. Pipeline DSP must be updated BEFORE `jack_client.start()`
3. SHM cleanup must not delete running JACK server files
4. bt-a2dp must be stopped before killing jackd

**Verify:**
```bash
# Check what JACK actually started with
ps aux | grep jackd | grep -v grep
# Expected: jackd -P 70 -d alsa -d hw:0,0 -r 48000 -p 512 -n 2 -i 2 -o 2
```

### Settings revert after restart

**Check:**
1. Does `config.yaml` contain `period:` and `rate:`?
2. Does `AudioConfig` load them from `config.yaml`?
3. Is the `latency_profile` property applying the overrides?

```bash
grep -E "period:|rate:|profile:" ~/.pedal/config.yaml
```

## References

- JACK documentation: https://jackaudio.org/faq/
- RPi 4B audio latency: https://wiki.linuxaudio.org/wiki/raspberrypi
- ALSA xrun_debug: https://www.alsa-project.org/wiki/XRUN_Debug
- Pi 4B CPU freq scaling: /sys/devices/system/cpu/cpu*/cpufreq/