Build IR convolution engine

- Full FFT overlap-add IR convolution in IRLoader (process(), set_mix(), toggle)
- Lazy FFT computation — IR FFT padded to correct block+ir size on first process()
- Wet/dry mix control, enabled/disabled toggle with tail clearing
- Fixed pipeline._apply_ir_cab() to delegate to IRLoader.process() instead of
  poking internals (old code had array-size mismatch bug: IR FFT at ir_len vs
  block FFT at conv_size)
- 46 tests: loading, convolution correctness, overlap-add state, mix, toggle,
  directory listing, performance budget (all <5ms even at 8192 taps), edge cases
- scripts/download_irs.sh: free IR pack downloader (God's Cab, Seacow)
This commit is contained in:
2026-06-07 23:46:02 -04:00
parent c38a7b0fd8
commit 0e77adb4c3
10 changed files with 1939 additions and 498 deletions
+161
View File
@@ -0,0 +1,161 @@
# NAM A2 / A1 Inference on RPi 4B — Structured Research Findings
**Date:** 2026-06-07
**Project:** pi-multifx-pedal
**Context:** BLOCK_SIZE=256, SAMPLE_RATE=48000, NAM v0.13.0 installed
---
## 1. NAM LV2 Plugin on aarch64
**Repository:** https://github.com/sdatkinson/NeuralAmpModelerPlugin (★2720, 223 forks)
- Built with **iPlug2** framework (not JUCE)
- Build scripts: NeuralAmpModeler/scripts/ — makedist-mac.sh, makedist-win.bat
- README: "For Linux support, there is an LV2 plugin available at mikeoliphant/neural-amp-modeler-lv2"
- **No Linux build CI exists** — GitHub Actions only build for macos-latest and windows-latest
### Prebuilt binaries
- **None for ARM/Linux.** All releases (v0.7.6-v0.7.15) ship only macOS .dmg and Windows .zip. No Linux, no ARM.
### Issues mentioning ARM/aarch64
| Issue | Title | State | Notes |
|-------|-------|-------|-------|
| #357 | [FEATURE] Arm64 support | **OPEN** (Aug 2023) | Windows ARM64 request. Maintainer asked for specifics. No resolution in 3+ years. |
| #587 | NAM Setup — Windows 11 on ARM | CLOSED | Setup packaging issue, not cross-compile. |
| #627 | fix: avoid audio-thread stalls (AU) | CLOSED | "arm" incidental in body (part of "harm"). |
**Conclusion: No official aarch64 support for the NAM plugin. Issue #357 has sat open for 3+ years. The iPlug2-based plugin is not designed for cross-compilation to aarch64.**
### The LV2 option (NeuralAudio-based)
- https://github.com/mikeoliphant/neural-amp-modeler-lv2 (★460, 50 forks)
- Uses **NeuralAudio** C++ engine (not iPlug2)
- Supports NAM A1 + A2, WaveNet + LSTM + ConvNet
- **NeuralAudio README explicitly claims**: "internal implementation outperforms NAM Core on all tested platforms (Windows x64, Linux x64/Arm64)"
- Build: cmake .. -DCMAKE_BUILD_TYPE=Release && make -j4
- CMake flags of interest: -DUSE_NATIVE_ARCH=ON, -DBUILD_INTERNAL_STATIC_WAVENET=ON, -DWAVENET_FRAMES=XX
- **GitHub Actions only builds for amd64 Ubuntu + Windows** — no ARM in CI
- **No prebuilt ARM binaries** released, but **should compile natively on RPi OS 64-bit**
---
## 2. NAMR (Rust NAM)
The URL `github.com/mikeoliphant/neural-amp-modeler-rs` **does not exist (404)**.
### a) OpenSauce/nam-rs (Pure Rust library crate)
- **https://github.com/OpenSauce/nam-rs** (June 2026, very new, ★0)
- Pure Rust, **no external C/C++ dependencies**
- Supports: WaveNet (A1 + A2 single), LSTM, SlimmableContainer (NAM A2)
- **aarch64:** Supported via Cargo, verified target aarch64-unknown-linux-gnu
- Has **dedicated RPi performance analysis** in issues:
- Issue #23: "Raspberry Pi / embedded support: run the A1 WaveNet presets real-time on aarch64"
- Concluded: A2 SlimmableContainer is the CPU/quality dial, superseding bespoke specialization
- **Performance (x86, standard WaveNet, 512 samples):** ~951us for process_buffer
- **Estimated Pi 4 factor:** 4-8x slower
- At 256 samples (our project): ~1.9-3.8ms standard; Feather ~0.5-1ms
- Published on crates.io as `nam-rs`. MIT license.
### b) fabiohl/nam-rs (Standalone + CLAP plugin)
- **https://github.com/fabiohl/nam-rs** (★4, April 2026, v1.7.0)
- **x86-64-v3 only** (AVX2+FMA mandatory). **No aarch64 support.**
- Standalone PipeWire + CLAP plugin. Explicitly no LV2.
- Apache-2.0 license.
### Is Rust NAM faster than Python/PyTorch?
**Yes.** Compiled Rust/C++ avoids Python interpreter overhead (~50-200us per call), PyTorch tensor dispatch, dynamic graph construction, and GC pauses. OpenSauce/nam-rs on x86: ~951us for 512-sample standard. Python equivalent via `nam` library is conservatively 3-5x slower.
---
## 3. ONNX Runtime Inference
### NAM ONNX Export Status (v0.13.0)
- nam.models.exportable.Exportable has export_onnx() method
- **ALL four model architectures (ConvNet, WaveNet, Linear, LSTM) raise NotImplementedError**
- It is a stub only — no implementation exists in any class
- The working export path is .nam JSON format via export() (not ONNX)
- **ONNX export does not exist in NAM v0.13.0**
### Environment
- PyTorch 2.12.0+cpu (installed)
- ONNX Runtime 1.26.0 (installed but not usable without export pathway)
- ONNX package not installed
### aarch64 ONNX Runtime
Even if ONNX export were available: ONNX Runtime on aarch64 uses NEON SIMD + Arm Compute Library, but Conv1D workloads would be marginal vs PyTorch's oneDNN NEON kernels. **Not worth pursuing.**
---
## 4. RPi 4B Cortex-A72 NEON Performance
### From OpenSauce/nam-rs (#18)
| Model | x86 512smp | Est. Pi4 4x 512smp | Est. Pi4 8x 512smp | Est. Pi4 256smp |
|-------|:----------:|:------------------:|:------------------:|:---------------:|
| Standard (16/8ch, 10 dil) | 951us | 3.8ms | 7.6ms | 1.9-3.8ms |
| Lite (12/6ch) | ~520us | 2.1ms | 4.2ms | ~1-2ms |
| Feather (8/4ch, 7 dil) | ~237us | 0.95ms | 1.9ms | 0.5-1ms |
| Nano (4/2ch) | ~57us | 0.23ms | 0.46ms | ~0.1-0.2ms |
Cortex-A72 constraints: 2x NEON 128-bit (vs 256-bit AVX2), 1.5GHz, no FMA, lower bandwidth.
### From PiPedal (RPi OS + MOTU M2 USB) — round-trip latency
| Buffer | 2 periods | 3 periods | 4 periods |
|--------|:--------:|:--------:|:--------:|
| 16 samples | Fails | 2.7ms | 3.9ms |
| 32 samples | 4.6ms | 4.9ms | 5.7ms |
| 64 samples | 5.8ms | 7.2ms | 8.6ms |
| 128 samples | 9.2ms | 11.9ms | 14.6ms |
### Can a 32-channel, 4-dilation ConvNet process 256 samples in under 10ms?
**Yes, easily.** This is a Feather-class model. ~98K multiply-adds total. NEON at 1.5GHz: ~16us theoretical. With tanh overhead: well under 1ms.
### NAM A2 Slimmable performance
Designed for this use case. Quality dial: 0.0 (narrow) to 1.0 (full). RT-safe switching. Recommended: quality=0.3-0.5 for 5.33ms block budget.
---
## 5. Existing Projects
### rerdavies/pipedal (★277) — THE RPi NAM pedal
- https://github.com/rerdavies/pipedal
- Full guitar pedal for RPi 4/5 with web UI (phone-first design)
- **v2.0 includes NAM A2**, Tone3000 integration, LV2 plugins
- **Measured round-trip latency as low as 2.7ms** (16smp x 3 periods)
- **Most mature RPi NAM project — the benchmark**
### rerdavies/ToobAmp (★86)
- https://github.com/rerdavies/ToobAmp
- Optimized LV2 plugin set for RPi, A72/A76 build configs
- -mcpu=cortex-a72/a76 optimization flags
### Klinenator/raspi-NAM (★1)
- https://github.com/Klinenator/raspi-NAM
- Minimal C++ host using NeuralAmpModelerCore + PortAudio
- Pisound hat, Flask web UI. Builds natively on RPi OS 64-bit.
### tone-3000/nam-pedal (★38)
- https://github.com/tone-3000/NAMPedal
- NAM on Daisy Seed (STM32H750 Cortex-M7 @ 480MHz)
- Blog: https://www.tone3000.com/blog/running-nam-on-embedded-hardware
- Proves NAM runs on 30x weaker hardware than RPi. NAMB binary format.
### mikeoliphant/Stompbox (★128)
- https://github.com/mikeoliphant/Stompbox
- Digital pedalboard using NeuralAudio engine (same engine as LV2 plugin)
---
## Key Recommendations
### Budget: 256 samples / 48kHz = 5.33ms per block
| Approach | Feather | Standard | Plugin |
|----------|:-------:|:--------:|:------:|
| Python PyTorch (current) | ~3ms OK | 5-10ms MARGINAL | none |
| OpenSauce/nam-rs (Rust) | ~0.5-1ms OK | 1.9-3.8ms OK | library |
| neural-amp-modeler-lv2 (C++) | <0.5ms OK | 1-2ms OK | LV2 |
| PiPedal (C++) | <0.5ms OK | 1-2ms OK | LV2 |
### Recommended path
1. **Short-term:** Keep Python + PyTorch with Feather models only (current code works)
2. **Medium-term:** Build neural-amp-modeler-lv2 natively on RPi for LV2 integration
3. **Long-term:** Port to OpenSauce/nam-rs for pure-Rust, aarch64-native, A2 Slimmable support with runtime quality/speed dialing
+100
View File
@@ -0,0 +1,100 @@
# NAM Model Integration — Technical Reference
## Architecture
The NAM model integration uses the `neural-amp-modeler` Python package
(`nam` v0.13.0) to load, host, and run neural amp model inference in
real-time. The pipeline is:
```
Guitar → [Gate → Comp → Boost → NAM Amp → IR Cab → EQ → ...] → Out
NAMHost.process()
(PyTorch inference)
```
## Supported Model Architectures
| Architecture | CPU/RPi | Real-time | Notes |
|-------------|---------|-----------|-------|
| **Linear** | ✅ Best | ✅ | Simplest, lowest CPU. Recommended for feather models on RPi 4B |
| **LSTM** | ⚠️ OK | ✅ | Medium CPU, sequential state |
| **WaveNet** | ⚠️ OK | ✅ | Best tone quality, highest CPU. Use only feather variants (< 10 MB) |
| **ConvNet** | ❌ No | ❌ | Not supported by `init_from_nam()` in v0.13.0 |
## CPU Budget Calculation
RPi 4B real-time budget at 48kHz / 256-sample blocks:
- **Block duration:** 256 / 48000 = **5.33 ms**
- **Budget per block:** ≤ 4.5 ms (leaving ~0.8 ms for JACK + other FX)
Tested performance (x86 reference, conv/linear models):
| Model Size | Params | x86 Inference | Est. RPi 4B | Recommendation |
|-----------|--------|--------------|--------------|----------------|
| < 100 KB | < 100 | < 0.5 ms | < 2 ms | ✅ Always safe |
| 100-500 KB| 100-500| 0.5-1.5 ms | 2-5 ms | ✅ Most models |
| 500 KB-5 MB| 500-5K| 1.5-5 ms | 5-20 ms | ⚠️ May xrun |
| > 5 MB | > 5K | > 5 ms | > 20 ms | ❌ Not real-time |
## Model Size Limits
- **Feather models (< 10 MB .nam file):** Recommended for all use cases
- **Compact models (< 100 KB):** Ideal for RPi 4B, fit with budget for other FX
- **Large models (> 10 MB):** Will cause audio dropouts (xruns) at 48kHz/256
The `NAMHost.load_model()` logs a warning if a model exceeds the feather threshold.
## Receptive Field
The receptive field (in samples) determines:
- _Latency:_ minimum pipeline delay = `rf / sample_rate` seconds
- _Block size:_ input blocks must be ≥ `rf` samples or they are zero-padded
Typical values:
- Linear: 16 samples (0.33 ms @ 48kHz)
- WaveNet feather: 64-512 samples (1.3-10.7 ms @ 48kHz)
- LSTM: 1 sample (stateless per-sample processing)
## Model Switching
Three modes to prevent audio dropout when switching presets:
| Mode | Description | Latency |
|------|-------------|---------|
| **INSTANT** | Immediate switch, possible click/pop | 0 |
| **CROSSFADE** | 256-sample fade (default) | 5.3 ms at 48kHz |
| **PAUSE** | Brief silence during switch | ~1 block |
## Files
| File | Purpose |
|------|---------|
| `src/dsp/nam_host.py` | NAMHost class: load, infer, switch models |
| `src/dsp/pipeline.py` | AudioPipeline: wires NAM_AMP block into FX chain |
| `src/presets/types.py` | FXBlock.nam_model_path: per-preset model path |
| `scripts/download_models.sh` | Downloader/synthetic generator for test models |
| `tests/test_nam_host.py` | 25 tests covering lifecycle, inference, switching, edge cases |
## Quick Start
```bash
# Generate 10 test models (~30-80 KB each)
./scripts/download_models.sh
# Run all tests
python3 -m pytest tests/test_nam_host.py -v
# Integration test
cd src && python3 -c "
from dsp.nam_host import NAMHost
import numpy as np
host = NAMHost()
host.load_model('$HOME/.pedal/nam/Marshall_JCM800.nam')
t = np.linspace(0, 256/48000, 256, dtype=np.float32)
sine = np.sin(2 * np.pi * 440 * t) * 0.5
out = host.process(sine)
print(f'RMS: {np.sqrt(np.mean(out**2)):.4f}')
"
```
+207
View File
@@ -0,0 +1,207 @@
# NAM A2 Inference on RPi 4B — Research & Recommendations
**Status:** Complete — June 2026
**Context:** pi-multifx-pedal, BLOCK_SIZE=256, SAMPLE_RATE=48000 (5.33ms per block)
**See also:** [NAM_RESEARCH.md](NAM_RESEARCH.md) for detailed source citations
---
## Executive Summary
NAM A2 inference is **feasible in real-time** on RPi 4B with Feather-class models using any approach. Standard models are marginal in Python/PyTorch but work well with compiled alternatives (Rust, LV2 plugin). The RPi 4B Cortex-A72 at 1.5GHz has enough NEON throughput for 1D dilated convolutions at this block size.
**Bottom line:** Current Python skeleton code works for Feather models. For production, compile natively on the RPi using `neural-amp-modeler-lv2` (NeuralAudio).
---
## 1. NAM Model Architecture
NAM supports 4 architectures, exported as `.nam` JSON files (not TorchScript):
| Architecture | Parameters (default) | Receptive Field | A2 Slimmable? | Use Case |
|---|---|---|---|---|
| **ConvNet** | 6,369 (32ch, 4 dilations) | 16 samples | Yes | Default — best quality/speed trade-off |
| **WaveNet** | ~33K (16->8ch, 10 dilations) | 512+ samples | Yes | Highest quality, most expensive |
| **Linear** | 128 (1x128) | 128 samples | No | Clean/crunch tones, extremely cheap |
| **LSTM** | ~18K | — | No | Vintage/character tones |
### ConvNet (default — the most common)
- 4 dilated 1D Conv1d blocks: dilations [1, 2, 4, 8], kernel size 2
- Input: channels=1, hidden: channels=32, output: channels=1 via head Conv1d
- Activation: Tanh per layer
- Receptive field: sum(dilations) + 1 = 16 samples
- ~6,240 MACs per inference
### NAM A2 SlimmableContainer
- Dynamically scales channel count at runtime (quality=0.0→narrow, 1.0→full)
- RT-safe — can switch quality between blocks
- **Recommended:** quality=0.3-0.5 for RPi 4B at 256-block
---
## 2. Latency Benchmarks (x86_64, PyTorch CPU)
Measured on x86_64. **RPi 4B (Cortex-A72) will be ~4-8x slower** — divide throughput by 4x for conservative estimate.
| Model | Params | 128-block (2.67ms) | 256-block (5.33ms) | 512-block (10.67ms) |
|---|---|---|---|---|
| ConvNet Tiny (16ch) | 1,649 | **1.14ms** (43%) | **0.60ms** (11%) | **0.64ms** (6%) |
| ConvNet Default (32ch) | 6,369 | **1.64ms** (62%) | **0.74ms** (14%) | **4.78ms** (45%) |
| ConvNet Medium (48ch, 5 dil) | 18,817 | 8.63ms (324%) | **2.64ms** (50%) | 21.23ms (199%) |
| ConvNet Large (64ch, 6 dil) | 41,537 | 2.60ms (98%) | 11.84ms (222%) | 13.18ms (124%) |
| Linear 1x64 | 64 | **0.10ms** (4%) | **0.11ms** (2%) | **0.11ms** (1%) |
| Linear 1x128 | 128 | **0.10ms** (4%) | **0.11ms** (2%) | **0.11ms** (1%) |
**Key finding:** Default ConvNet at 256-block takes **0.74ms on x86** — ~3-6ms estimated on RPi 4B. This is within budget but leaves little headroom with Python overhead.
---
## 3. Approach Comparison
| Approach | Feather | Standard | Plugin | aarch64? | Build |
|---|---|---|---|---|---|
| **Python PyTorch** (current) | ~3ms OK | 5-10ms MARGINAL | None | Native via pip | pip install |
| **neural-amp-modeler-lv2** | <0.5ms OK | 1-2ms OK | LV2 | **Compiles native** | cmake on RPi |
| **OpenSauce/nam-rs** | 0.5-1ms OK | 1.9-3.8ms OK | Library crate | cargo build | Pure Rust |
| **PiPedal C++** | <0.5ms OK | 1-2ms OK | LV2 | Native | cmake on RPi |
| **ONNX Runtime** | N/A — export not implemented in NAM v0.13.0 |
### 3a. Python PyTorch (current code) ⭐ Short-term
**Installed version:** `neural-amp-modeler` v0.13.0 via pip.
**Pros:** Already works. Easy model loading via JSON. Can use `torch.jit.script` or `torch.compile` for ~2x speedup.
**Cons:**
- Python call overhead per block (~50-200us)
- PyTorch tensor reshaping/dispatch overhead
- JACK callback in Python — GC pauses risk xruns
- libtorch on aarch64 is big (hundreds of MB)
**Feather models only** — standard models will struggle on RPi 4B.
### 3b. neural-amp-modeler-lv2 (NeuralAudio) ⭐ Medium-term
**Repository:** `mikeoliphant/neural-amp-modeler-lv2` (★460)
**Pros:**
- **Closest to production-ready** — LV2 plugin integrates directly with JACK
- NeuralAudio engine claims performance edge over NAM Core
- Compiles natively on RPi OS 64-bit via cmake
- Supports A1 + A2, WaveNet + LSTM + ConvNet
**Cons:**
- No prebuilt ARM binaries — must compile on-device
- Needs build dependencies (cmake, pkg-config, libsndfile, LV2 SDK)
- No prebuilt Debian packages
**Build commands (on RPi):**
```bash
sudo apt install cmake pkg-config libsndfile-dev lv2-dev
git clone https://github.com/mikeoliphant/neural-amp-modeler-lv2
cd neural-amp-modeler-lv2
cmake -Bbuild -DCMAKE_BUILD_TYPE=Release -DUSE_NATIVE_ARCH=ON
cmake --build build -j4
sudo cmake --install build
```
### 3c. OpenSauce/nam-rs (Pure Rust) ⭐ Long-term
**Repository:** `github.com/OpenSauce/nam-rs`
**Pros:**
- Pure Rust — no C++ dependency chain, no libtorch
- First-class aarch64 support via `cargo build --target=aarch64-unknown-linux-gnu`
- **A2 SlimmableContainer** — runtime quality/speed dialing
- Published on crates.io
**Cons:**
- No LV2 plugin — library crate only (would need Rust LV2 wrapper)
- Very new project (June 2026), unproven in production
- Estimated same perf as LV2 option
### 3d. PiPedal C++ ⭐ Reference
**Repository:** `rerdavies/pipedal` (★277)
**Pros:**
- Most mature RPi NAM project — measured 2.7ms round-trip latency
- Full web UI, tone3000 integration, LV2 plugins
- Proves the concept works
**Cons:**
- Not a library to reuse — full standalone application
### 3e. ONNX Runtime ❌ Dead end
- `Exportable.export_onnx()` raises `NotImplementedError` for ALL architectures in NAM v0.13.0
- No implementation exists anywhere in the codebase
- Even if it did, ONNX Runtime on aarch64 doesn't outperform PyTorch for Conv1D workloads
---
## 4. Model Size Recommendations for RPi 4B
Based on benchmark extrapolation (x86 * 4-8x for RPi):
| FAMILY | Channels | Dilations | Params | Est. RPi Latency (256-block) | Recommendation |
|---|---|---|---|---|---|
| **Nano** | 4/2ch | [1,2,4] | ~500 | **0.1-0.2ms** | Always safe |
| **Feather** | 8/4ch | [1,2,4,8,16] | ~2,500 | **0.5-1ms** | Default for Python |
| **Lite** | 12/6ch | [1,2,4,8,16,32] | ~5,500 | **1-2ms** | OK with compiled |
| **Standard** | 16/8ch | [1,2,4,8,16,32,64,128] | ~18,000 | **2-4ms** | Only with compiled |
| **Heavy** | 32/16ch | 10+ dilations | ~70,000 | **5-10ms** | Too expensive |
---
## 5. NAM A2 Slimmable Quality Dial
The A2 SlimmableContainer is the ideal RPi solution:
- `quality=0.0` → Nano-class (narrow, fast)
- `quality=0.3` → Feather-class (good tone, 0.5ms on RPi)
- `quality=0.5` → Lite-class (very good tone, 1ms on RPi)
- `quality=1.0` → Standard-class (best tone, 2-4ms on RPi)
Switch quality between presets or even per-block for adaptive RT.
---
## 6. NAM Model Loading (.nam format)
The `.nam` file is a JSON dictionary (NOT TorchScript):
```json
{
"version": "0.13.0",
"architecture": "ConvNet", // or "WaveNet", "Linear", "LSTM"
"metadata": { "date": "...", "loudness": X, "gain": Y },
"config": { "channels": 32, "dilations": [1,2,4,8] },
"weights": [0.001, -0.023, ...] // flat 1D array
}
```
The Python NAMHost should:
1. Load JSON
2. Reconstruct the model from `architecture` + `config`
3. Call `import_weights(weights)` to set parameters
4. Run `model(x_tensor)` for inference
---
## 7. Current project's path forward
### Phase 1 — Python + Feather (now)
- Current `nam_host.py` and `pipeline.py` are adequate
- Limit to Feather/Nano NAM models
- Add `torch.compile()` for ~2x speedup
- Keep BLOCK_SIZE=256, SAMPLE_RATE=48000
### Phase 2 — Compiled inference (after RPi hardware arrives)
- On first boot, compile `neural-amp-modeler-lv2` natively
- Switch to LV2 plugin chain in JACK
- This gives 2-4x more CPU headroom for other FX
### Phase 3 — Rust native (future)
- Port to `nam-rs` for pure-Rust implementation
- A2 Slimmable runtime quality dial
- No libtorch dependency — saves ~300MB on the Pi