research: LLM strategy deliverable — hybrid local-first recommendation

2026-07-04 03:54:02 -04:00
parent 1479e161ab
commit 85d460e749
1 changed files with 347 additions and 0 deletions
@@ -0,0 +1,347 @@
+# LLM Strategy — Hermes Portable Rescue
+
+**Task:** T2 — research:llm-strat
+**Author:** Ashley (assistant)
+**Date:** 2026-07-04
+**Status:** Complete — ready for T3 (workstream:arch-decision)
+
+---
+
+## Executive Summary
+
+**Recommended approach: Hybrid Local-First strategy**
+
+| Layer | Model | When | Storage | RAM |
+|-------|-------|------|---------|-----|
+| **Primary** | Qwen2.5-7B-Instruct Q4_K_M (4.4 GB) | Always, offline-first | USB | ~7 GB |
+| **Lightweight fallback** | Qwen2.5-3B-Instruct Q4_K_M (2.0 GB) | Target < 8 GB RAM | USB | ~4 GB |
+| **API upgrade** | Cloud API (OpenCode Go / DeepSeek-V4) | Network available, complex reasoning | — | — |
+
+**Total USB storage for models:** 4.4 GB (primary) + 2.0 GB (fallback) = 6.4 GB
+**Recommended USB drive size:** 64 GB+
+
+---
+
+## 1. Options Evaluated
+
+### 1A — Local GGUF Model (Offline-only)
+
+**How it works:** Ship a pre-quantized GGUF model on the USB drive. On boot, launch `llama-server` (single binary, ~12 MB statically compiled) which provides an OpenAI-compatible API on localhost. Hermes agent connects to it as its provider.
+
+**Best for:**
+- PCs with no working network (broken WiFi driver, dead NIC, paywalled captive portal)
+- Air-gapped / sensitive environments
+- Consistent, predictable performance
+
+**LLM backpack (llama-server):** 12 MB static binary, no Python deps, no shared libraries.
+
+**Risks:**
+- Slower than API (10-40 tok/s on CPU vs 50-200+ tok/s via API)
+- Model size matters — must fit in available RAM after OS overhead
+- Cannot use large models (70B is out of reach on consumer hardware)
+
+### 1B — API-based (Cloud-only)
+
+**How it works:** The rescue agent requires a working internet connection and calls a cloud API (OpenCode Go, OpenRouter, Anthropic, etc.) for every inference.
+
+**Best for:**
+- Complex multi-factor reasoning (BSOD chain analysis, rare hardware issues)
+- Access to state-of-the-art models (Claude, GPT-4, DeepSeek-V4-Pro)
+- Minimal USB storage overhead (no model files)
+
+**Risks:**
+- **Crippling failure mode** — The most common reason for a rescue USB is a PC with broken network. If the agent can't think without the API, it's useless when you need it most.
+- Latency on spotty/paywalled connections
+- Privacy — diagnostic data leaves the target machine
+- API cost per session
+
+### 1C — Hybrid (Local-First, API-Enhanced)
+
+**How it works:** Ship both a local GGUF model and an API config. Default to local inference. If a working internet connection is detected AND the local model's output is low-confidence (or the task is complex enough), fall through to a cloud API call for the specific step.
+
+**Best for:**
+- Everything — offline-capable by default, enhanced when possible
+- Graceful degradation: works on a PC with a fried NIC, gets *better* on a PC with working WiFi
+
+**Risks:**
+- Slightly larger USB image (model files + API config)
+- Two code paths to test and maintain
+- Need a heuristic for "when to escalate to API"
+
+---
+
+## 2. Model Comparison
+
+### Candidate Models for Local Inference
+
+| Model | Size (params) | Q4_K_M Size | RAM Needed | Speed (CPU) | Technical Reasoning |
+|-------|--------------|-------------|------------|-------------|-------------------|
+| **Qwen2.5-7B-Instruct** | 7.6B | **4.4 GB** | ~7 GB | 15-25 tok/s | Excellent — strong instruction following |
+| Qwen2.5-Coder-7B-Instruct | 7.6B | 4.4 GB | ~7 GB | 15-25 tok/s | Great for code, overfit for general repair |
+| Llama-3.2-3B-Instruct | 3.2B | **1.9 GB** | ~4 GB | 25-40 tok/s | Good for simple diagnostics, weak at chains |
+| Qwen2.5-3B-Instruct | 3.1B | **2.0 GB** | ~4 GB | 25-40 tok/s | Good-small model, decent reasoning |
+| Phi-3.5-mini-Instruct | 3.8B | 2.2 GB | ~5 GB | 20-35 tok/s | Decent technical, smaller tokenizer |
+| Mistral-7B-v0.3 | 7.3B | 4.4 GB | ~7 GB | 13-23 tok/s | Strong general, slower than Qwen |
+| Gemma-2-9B | 9.2B | 5.5 GB | ~8 GB | 10-18 tok/s | Very strong, but needs more RAM |
+
+### Recommended: Qwen2.5-7B-Instruct Q4_K_M
+
+**Why this model:**
+1. **Best quality-to-size ratio** — Tops the Open LLM Leaderboard in the 7B class, with strong instruction following and reasoning. Perfect for parsing minidump stop codes, SMART attribute values, and multi-step diagnostic chains.
+2. **Qwen2.5 tokenizer** handles technical English and hex dumps efficiently.
+3. **Q4_K_M** is the sweet spot — only 1.7% perplexity loss vs FP16, 4x smaller than FP16.
+4. **Single file** — 4.36 GB for Q4_K_M, no shards to manage.
+
+**Why NOT Qwen2.5-Coder-7B:**
+- Overfitted for code generation. A rescue agent needs to reason about stop codes, hardware specs, and repair procedures — not generate code. The general instruct variant handles these better.
+
+### Lightweight Fallback: Qwen2.5-3B-Instruct Q4_K_M
+
+- 1.96 GB on disk, runs in ~4 GB RAM
+- Covers PCs with only 8 GB total RAM (OS overhead + Hermes leaves ~4-5 GB for the model)
+- Speedy: 25-40 tok/s on modern CPU
+
+### RAM Calculation for Target PCs
+
+| Target PC RAM | OS Overhead (Live Linux) | Available for Model | Can Run 7B Q4_K_M? | Can Run 3B Q4_K_M? |
+|--------------|--------------------------|-------------------|--------------------|--------------------|
+| 4 GB | ~1-1.5 GB | ~2.5-3 GB | No | **Tight** |
+| 8 GB | ~1-1.5 GB | ~6.5-7 GB | **Yes** | Yes |
+| 16 GB | ~1-1.5 GB | ~14.5-15 GB | Yes | Yes |
+| 32 GB | ~1-1.5 GB | ~30 GB | Yes | Yes |
+
+**Conclusion:** The 7B Q4_K_M model requires at least 8 GB of target PC RAM. This covers the vast majority of consumer and gaming PCs from the last 7 years. For low-RAM machines (< 8 GB), fall back to the 3B model.
+
+---
+
+## 3. Boot Environment Implications
+
+The LLM strategy is **tightly coupled** with the boot environment (T1 — research:boot-env).
+
+| Boot Env | LLM Viability | Notes |
+|----------|--------------|-------|
+| **Linux live** (Alpine, Arch, custom) | ✅ **Excellent** | Native llama.cpp binary, full RAM access, Python with pip, simple process management |
+| **WinPE** | ❌ **Impractical** | No native llama.cpp without MSVC runtime. No Python unless embedded. WinPE is heavily stripped (no .NET, no VC++ redist). Running LLM inference requires significant hacks. |
+| **Ventoy + Linux ISO** | ✅ **Excellent** | Same as Linux live — llama.cpp + Python work natively |
+
+**The LLM strategy effectively requires a Linux-based boot environment.** If the boot-env research (T1) concludes WinPE must be used, this strategy must be revised significantly (likely API-only with offline fallback being extremely limited diagnostic scripts without LLM).
+
+**Assuming Linux live environment:**
+
+```
+Boot → Start llama-server in background → Start Hermes agent → Hermes connects to localhost:8080
+
+[llama-server]                   [Hermes Agent]
+  ┌──────────────┐                ┌──────────────┐
+  │ llama-server  │←──localhost──│ Hermes core   │
+  │ --port 8080   │    :8080/v1  │ provider:     │
+  │ --ctx-size 4096│              │   base_url:   │
+  │ model.gguf     │              │   http://     │
+  └──────────────┘                │   localhost    │
+                                  │   :8080/v1    │
+                                  └──────────────┘
+```
+
+---
+
+## 4. Hermes Provider Configuration
+
+Hermes supports **any OpenAI-compatible API** as a provider by setting `model.base_url` in config.yaml. For the rescue environment, the portable config would be:
+
+```yaml
+# ~/.hermes/config.yaml (rescue profile)
+model:
+  # If local model is running:
+  base_url: http://localhost:8080/v1
+  api_key: ""            # or "sk-no-key-required"
+  default: qwen2.5-7b-instruct-q4_k_m
+  provider: ""           # base_url overrides provider
+
+  # If API fallback (network available):
+  # base_url: https://opencode.ai/zen/go/v1/
+  # api_key: ${OPENCODE_GO_API_KEY}
+  # default: deepseek-v4-flash
+```
+
+**Local inference stack:**
+1. `llama-server` — single static binary (~12 MB), started as systemd service or background process
+2. Launched with: `llama-server -m /usb/hermes/models/qwen2.5-7b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx-size 4096 --n-gpu-layers 0 --no-mmap`
+3. Hermes agent detects the API is available, sets base_url, and works normally
+
+**Key flag: `--no-mmap`** — Prevents memory-mapping the model file, important when running from a USB stick (mmap on slow USB can cause stuttering). Falls back to read-ahead loading.
+
+---
+
+## 5. Storage Budget on USB
+
+| Component | Size | Notes |
+|-----------|------|-------|
+| Linux live OS + kernel | ~500 MB-1.5 GB | Alpine minimal ~200 MB, Arch ~800 MB, custom ~1 GB |
+| llama-server binary | ~12 MB | Static build, no deps |
+| Qwen2.5-7B Q4_K_M model | **4.4 GB** | Primary model |
+| Qwen2.5-3B Q4_K_M model | **2.0 GB** | Fallback (optional) |
+| Hermes agent (Python + venv) | ~200-400 MB | Minimal venv, hermes core |
+| Diagnostic tools | ~100-300 MB | smartmontools, dmidecode, stress-ng, MemTest86, dd, etc. |
+| Docker image cache | ~0 MB optional | Running native binaries preferred |
+| **Total (with both models)** | **~7.5-9 GB** | |
+
+**All models fit comfortably on a 32 GB USB stick.** For 64 GB+ sticks, consider adding additional quantizations or a larger model (e.g., Qwen2.5-14B Q4_K_M at ~8 GB for extra reasoning capability).
+
+### USB Speed Impact
+
+- USB 3.0 (5 Gbps) → ~500 MB/s theoretical
+- USB 3.1 Gen 2 (10 Gbps) → ~1 GB/s
+- USB 2.0 (480 Mbps) → ~60 MB/s practical
+
+Model loading time at USB 3.0: ~9 seconds for 4.4 GB
+Model loading at USB 2.0: ~75 seconds (annoying but acceptable — boot once per session)
+
+**Recommendation:** Use `--no-mmap` flag with llama-server to avoid live USB I/O during inference. The model loads into RAM once at startup and operates entirely from RAM afterwards.
+
+---
+
+## 6. Speed & Latency Analysis
+
+| Scenario | Task | Local 7B Q4_K_M | API (DeepSeek-V4-Flash) |
+|----------|------|-----------------|------------------------|
+| First token | Cold start | ~1-3 sec | ~0.5-2 sec |
+| BSOD analysis | 200 tokens | ~10 sec | ~3-5 sec |
+| SMART interpretation | 400 tokens | ~20 sec | ~5-8 sec |
+| Repair plan | 800 tokens | ~35 sec | ~8-12 sec |
+
+**Local inference is 2-5x slower than API** but still fast enough for a diagnostic workflow where the user waits for results anyway. BSOD analysis at 10 seconds is perfectly acceptable — the alternative is the user manually Googling stop codes.
+
+---
+
+## 7. Offline Capability Assessment
+
+| Feature | Local GGUF | API-only | Hybrid |
+|---------|-----------|----------|--------|
+| Boot with dead NIC | ✅ | ❌ | ✅ |
+| Boot in paywalled WiFi | ✅ | ❌ | ✅ |
+| Boot in air-gapped env | ✅ | ❌ | ✅ |
+| Complex reasoning | ⚠️ (7B is decent) | ✅ | ✅ (API when available) |
+| BSOD analysis | ✅ | ✅ | ✅ |
+| SMART disk parsing | ✅ | ✅ | ✅ |
+| Driver download | ❌ (needs net) | ✅ | ⚠️ (net required anyway) |
+| Backup to cloud | ❌ (needs net) | ✅ | ⚠️ (net required anyway) |
+
+**Key insight:** The hybrid strategy handles the critical "PC won't boot / broken NIC" scenario that's the most common use case for a rescue USB. Both local and hybrid work offline for diagnostics. The only features that genuinely need a network (driver downloads, cloud backup) require it regardless of LLM strategy.
+
+---
+
+## 8. Model Download & Preparation
+
+Models are downloaded during the build phase (not at runtime):
+
+```bash
+# Build script: fetch models
+mkdir -p build/models/
+
+# Primary model — Qwen2.5-7B-Instruct Q4_K_M
+wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q4_k_m.gguf \
+  -O build/models/qwen2.5-7b-instruct-q4_k_m.gguf
+
+# Fallback model — Qwen2.5-3B-Instruct Q4_K_M
+wget https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.gguf \
+  -O build/models/qwen2.5-3b-instruct-q4_k_m.gguf
+
+# llama-server binary (Linux x86_64 static)
+wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server \
+  -O build/tools/llama-server
+chmod +x build/tools/llama-server
+```
+
+**Licensing:** Qwen2.5 models use the Apache 2.0 license — free for commercial and personal use.
+
+---
+
+## 9. Implementation Recommendation
+
+### Phase 1: Prototype (this sprint)
+
+1. **Download** Qwen2.5-7B-Instruct Q4_K_M (4.4 GB) to the NAS project directory
+2. **Build** a minimal Hermes profile ("rescue") with local provider config
+3. **Package** llama-server binary
+4. **Write** a startup script (launch-llm.sh) that:
+   - Probes available RAM
+   - Decides 7B vs 3B model based on free RAM
+   - Launches llama-server with appropriate flags
+   - Waits for API readiness
+   - Launches Hermes agent with local provider config
+5. **Test** locally on Shawn's host
+
+### Phase 2: Integration (post-arch-decision)
+
+1. Integrate startup script into the bootable USB image
+2. Wire model selection into the hardware inventory module (T5)
+3. Add API config template for fallback provider
+
+### Phase 3: Polish
+
+1. Benchmark model quality on real BSOD dumps
+2. Tune prompt templates for diagnostic tasks
+3. Optimize context window usage (shorter contexts = faster on local)
+4. Consider Knowledge Distillation: fine-tune a smaller model on repair-centric data
+
+---
+
+## 10. Risk Register
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|-----------|--------|------------|
+| Target PC has < 8 GB RAM | Medium | High (7B won't load) | Ship 3B fallback model |
+| USB 2.0 port (slow loading) | High (old PCs) | Medium (75s load time) | Show progress indicator, use --no-mmap |
+| Qwen2.5 underperforms on diagnostics | Low | Medium | Test on real dumps, fall back to API |
+| Boot env is WinPE (no llama.cpp) | TBD from T1 | Critical | Pivot to API-only or switch boot env |
+| Network available but slow | Medium | Low | Local handles all diagnostics; API only for complex |
+| Model licensing changes | Low | Medium | Apache 2.0 is permissive and stable |
+
+---
+
+## Appendix A: Quick-Reference Commands
+
+```bash
+# Start local LLM server
+llama-server \
+  -m /usb/hermes/models/qwen2.5-7b-instruct-q4_k_m.gguf \
+  --host 127.0.0.1 \
+  --port 8080 \
+  --ctx-size 4096 \
+  --n-gpu-layers 0 \
+  --no-mmap \
+  --flash-attn \
+  --threads $(nproc)
+
+# Verify it's running
+curl http://localhost:8080/v1/models
+
+# Test inference
+curl http://localhost:8080/v1/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "prompt": "What does BSOD 0x0000001A (MEMORY_MANAGEMENT) indicate?",
+    "model": "qwen2.5-7b-instruct-q4_k_m",
+    "max_tokens": 200,
+    "temperature": 0.1
+  }'
+
+# Hermes provider config (for rescue profile)
+# ~/.hermes/profiles/rescue/config.yaml:
+# model:
+#   base_url: http://localhost:8080/v1
+#   api_key: ""
+#   default: qwen2.5-7b-instruct-q4_k_m
+```
+
+## Appendix B: Automatic Model Selection Script (pseudocode)
+
+```
+1. Check target PC RAM: free -g | grep Mem
+2. If total RAM >= 8 GB → use 7B Q4_K_M (~7 GB needed)
+3. If total RAM >= 4 GB → use 3B Q4_K_M (~4 GB needed)
+4. If total RAM < 4 GB → fallback mode: API-only IF network, else limited scripted diagnostics
+5. Launch llama-server with selected model
+6. Wait for healthy endpoint
+7. Launch Hermes agent with local provider config
+```