diff --git a/research/llm-strategy.md b/research/llm-strategy.md new file mode 100644 index 0000000..f392750 --- /dev/null +++ b/research/llm-strategy.md @@ -0,0 +1,347 @@ +# LLM Strategy — Hermes Portable Rescue + +**Task:** T2 — research:llm-strat +**Author:** Ashley (assistant) +**Date:** 2026-07-04 +**Status:** Complete — ready for T3 (workstream:arch-decision) + +--- + +## Executive Summary + +**Recommended approach: Hybrid Local-First strategy** + +| Layer | Model | When | Storage | RAM | +|-------|-------|------|---------|-----| +| **Primary** | Qwen2.5-7B-Instruct Q4_K_M (4.4 GB) | Always, offline-first | USB | ~7 GB | +| **Lightweight fallback** | Qwen2.5-3B-Instruct Q4_K_M (2.0 GB) | Target < 8 GB RAM | USB | ~4 GB | +| **API upgrade** | Cloud API (OpenCode Go / DeepSeek-V4) | Network available, complex reasoning | — | — | + +**Total USB storage for models:** 4.4 GB (primary) + 2.0 GB (fallback) = 6.4 GB +**Recommended USB drive size:** 64 GB+ + +--- + +## 1. Options Evaluated + +### 1A — Local GGUF Model (Offline-only) + +**How it works:** Ship a pre-quantized GGUF model on the USB drive. On boot, launch `llama-server` (single binary, ~12 MB statically compiled) which provides an OpenAI-compatible API on localhost. Hermes agent connects to it as its provider. + +**Best for:** +- PCs with no working network (broken WiFi driver, dead NIC, paywalled captive portal) +- Air-gapped / sensitive environments +- Consistent, predictable performance + +**LLM backpack (llama-server):** 12 MB static binary, no Python deps, no shared libraries. + +**Risks:** +- Slower than API (10-40 tok/s on CPU vs 50-200+ tok/s via API) +- Model size matters — must fit in available RAM after OS overhead +- Cannot use large models (70B is out of reach on consumer hardware) + +### 1B — API-based (Cloud-only) + +**How it works:** The rescue agent requires a working internet connection and calls a cloud API (OpenCode Go, OpenRouter, Anthropic, etc.) for every inference. + +**Best for:** +- Complex multi-factor reasoning (BSOD chain analysis, rare hardware issues) +- Access to state-of-the-art models (Claude, GPT-4, DeepSeek-V4-Pro) +- Minimal USB storage overhead (no model files) + +**Risks:** +- **Crippling failure mode** — The most common reason for a rescue USB is a PC with broken network. If the agent can't think without the API, it's useless when you need it most. +- Latency on spotty/paywalled connections +- Privacy — diagnostic data leaves the target machine +- API cost per session + +### 1C — Hybrid (Local-First, API-Enhanced) + +**How it works:** Ship both a local GGUF model and an API config. Default to local inference. If a working internet connection is detected AND the local model's output is low-confidence (or the task is complex enough), fall through to a cloud API call for the specific step. + +**Best for:** +- Everything — offline-capable by default, enhanced when possible +- Graceful degradation: works on a PC with a fried NIC, gets *better* on a PC with working WiFi + +**Risks:** +- Slightly larger USB image (model files + API config) +- Two code paths to test and maintain +- Need a heuristic for "when to escalate to API" + +--- + +## 2. Model Comparison + +### Candidate Models for Local Inference + +| Model | Size (params) | Q4_K_M Size | RAM Needed | Speed (CPU) | Technical Reasoning | +|-------|--------------|-------------|------------|-------------|-------------------| +| **Qwen2.5-7B-Instruct** | 7.6B | **4.4 GB** | ~7 GB | 15-25 tok/s | Excellent — strong instruction following | +| Qwen2.5-Coder-7B-Instruct | 7.6B | 4.4 GB | ~7 GB | 15-25 tok/s | Great for code, overfit for general repair | +| Llama-3.2-3B-Instruct | 3.2B | **1.9 GB** | ~4 GB | 25-40 tok/s | Good for simple diagnostics, weak at chains | +| Qwen2.5-3B-Instruct | 3.1B | **2.0 GB** | ~4 GB | 25-40 tok/s | Good-small model, decent reasoning | +| Phi-3.5-mini-Instruct | 3.8B | 2.2 GB | ~5 GB | 20-35 tok/s | Decent technical, smaller tokenizer | +| Mistral-7B-v0.3 | 7.3B | 4.4 GB | ~7 GB | 13-23 tok/s | Strong general, slower than Qwen | +| Gemma-2-9B | 9.2B | 5.5 GB | ~8 GB | 10-18 tok/s | Very strong, but needs more RAM | + +### Recommended: Qwen2.5-7B-Instruct Q4_K_M + +**Why this model:** +1. **Best quality-to-size ratio** — Tops the Open LLM Leaderboard in the 7B class, with strong instruction following and reasoning. Perfect for parsing minidump stop codes, SMART attribute values, and multi-step diagnostic chains. +2. **Qwen2.5 tokenizer** handles technical English and hex dumps efficiently. +3. **Q4_K_M** is the sweet spot — only 1.7% perplexity loss vs FP16, 4x smaller than FP16. +4. **Single file** — 4.36 GB for Q4_K_M, no shards to manage. + +**Why NOT Qwen2.5-Coder-7B:** +- Overfitted for code generation. A rescue agent needs to reason about stop codes, hardware specs, and repair procedures — not generate code. The general instruct variant handles these better. + +### Lightweight Fallback: Qwen2.5-3B-Instruct Q4_K_M + +- 1.96 GB on disk, runs in ~4 GB RAM +- Covers PCs with only 8 GB total RAM (OS overhead + Hermes leaves ~4-5 GB for the model) +- Speedy: 25-40 tok/s on modern CPU + +### RAM Calculation for Target PCs + +| Target PC RAM | OS Overhead (Live Linux) | Available for Model | Can Run 7B Q4_K_M? | Can Run 3B Q4_K_M? | +|--------------|--------------------------|-------------------|--------------------|--------------------| +| 4 GB | ~1-1.5 GB | ~2.5-3 GB | No | **Tight** | +| 8 GB | ~1-1.5 GB | ~6.5-7 GB | **Yes** | Yes | +| 16 GB | ~1-1.5 GB | ~14.5-15 GB | Yes | Yes | +| 32 GB | ~1-1.5 GB | ~30 GB | Yes | Yes | + +**Conclusion:** The 7B Q4_K_M model requires at least 8 GB of target PC RAM. This covers the vast majority of consumer and gaming PCs from the last 7 years. For low-RAM machines (< 8 GB), fall back to the 3B model. + +--- + +## 3. Boot Environment Implications + +The LLM strategy is **tightly coupled** with the boot environment (T1 — research:boot-env). + +| Boot Env | LLM Viability | Notes | +|----------|--------------|-------| +| **Linux live** (Alpine, Arch, custom) | ✅ **Excellent** | Native llama.cpp binary, full RAM access, Python with pip, simple process management | +| **WinPE** | ❌ **Impractical** | No native llama.cpp without MSVC runtime. No Python unless embedded. WinPE is heavily stripped (no .NET, no VC++ redist). Running LLM inference requires significant hacks. | +| **Ventoy + Linux ISO** | ✅ **Excellent** | Same as Linux live — llama.cpp + Python work natively | + +**The LLM strategy effectively requires a Linux-based boot environment.** If the boot-env research (T1) concludes WinPE must be used, this strategy must be revised significantly (likely API-only with offline fallback being extremely limited diagnostic scripts without LLM). + +**Assuming Linux live environment:** + +``` +Boot → Start llama-server in background → Start Hermes agent → Hermes connects to localhost:8080 + +[llama-server] [Hermes Agent] + ┌──────────────┐ ┌──────────────┐ + │ llama-server │←──localhost──│ Hermes core │ + │ --port 8080 │ :8080/v1 │ provider: │ + │ --ctx-size 4096│ │ base_url: │ + │ model.gguf │ │ http:// │ + └──────────────┘ │ localhost │ + │ :8080/v1 │ + └──────────────┘ +``` + +--- + +## 4. Hermes Provider Configuration + +Hermes supports **any OpenAI-compatible API** as a provider by setting `model.base_url` in config.yaml. For the rescue environment, the portable config would be: + +```yaml +# ~/.hermes/config.yaml (rescue profile) +model: + # If local model is running: + base_url: http://localhost:8080/v1 + api_key: "" # or "sk-no-key-required" + default: qwen2.5-7b-instruct-q4_k_m + provider: "" # base_url overrides provider + + # If API fallback (network available): + # base_url: https://opencode.ai/zen/go/v1/ + # api_key: ${OPENCODE_GO_API_KEY} + # default: deepseek-v4-flash +``` + +**Local inference stack:** +1. `llama-server` — single static binary (~12 MB), started as systemd service or background process +2. Launched with: `llama-server -m /usb/hermes/models/qwen2.5-7b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx-size 4096 --n-gpu-layers 0 --no-mmap` +3. Hermes agent detects the API is available, sets base_url, and works normally + +**Key flag: `--no-mmap`** — Prevents memory-mapping the model file, important when running from a USB stick (mmap on slow USB can cause stuttering). Falls back to read-ahead loading. + +--- + +## 5. Storage Budget on USB + +| Component | Size | Notes | +|-----------|------|-------| +| Linux live OS + kernel | ~500 MB-1.5 GB | Alpine minimal ~200 MB, Arch ~800 MB, custom ~1 GB | +| llama-server binary | ~12 MB | Static build, no deps | +| Qwen2.5-7B Q4_K_M model | **4.4 GB** | Primary model | +| Qwen2.5-3B Q4_K_M model | **2.0 GB** | Fallback (optional) | +| Hermes agent (Python + venv) | ~200-400 MB | Minimal venv, hermes core | +| Diagnostic tools | ~100-300 MB | smartmontools, dmidecode, stress-ng, MemTest86, dd, etc. | +| Docker image cache | ~0 MB optional | Running native binaries preferred | +| **Total (with both models)** | **~7.5-9 GB** | | + +**All models fit comfortably on a 32 GB USB stick.** For 64 GB+ sticks, consider adding additional quantizations or a larger model (e.g., Qwen2.5-14B Q4_K_M at ~8 GB for extra reasoning capability). + +### USB Speed Impact + +- USB 3.0 (5 Gbps) → ~500 MB/s theoretical +- USB 3.1 Gen 2 (10 Gbps) → ~1 GB/s +- USB 2.0 (480 Mbps) → ~60 MB/s practical + +Model loading time at USB 3.0: ~9 seconds for 4.4 GB +Model loading at USB 2.0: ~75 seconds (annoying but acceptable — boot once per session) + +**Recommendation:** Use `--no-mmap` flag with llama-server to avoid live USB I/O during inference. The model loads into RAM once at startup and operates entirely from RAM afterwards. + +--- + +## 6. Speed & Latency Analysis + +| Scenario | Task | Local 7B Q4_K_M | API (DeepSeek-V4-Flash) | +|----------|------|-----------------|------------------------| +| First token | Cold start | ~1-3 sec | ~0.5-2 sec | +| BSOD analysis | 200 tokens | ~10 sec | ~3-5 sec | +| SMART interpretation | 400 tokens | ~20 sec | ~5-8 sec | +| Repair plan | 800 tokens | ~35 sec | ~8-12 sec | + +**Local inference is 2-5x slower than API** but still fast enough for a diagnostic workflow where the user waits for results anyway. BSOD analysis at 10 seconds is perfectly acceptable — the alternative is the user manually Googling stop codes. + +--- + +## 7. Offline Capability Assessment + +| Feature | Local GGUF | API-only | Hybrid | +|---------|-----------|----------|--------| +| Boot with dead NIC | ✅ | ❌ | ✅ | +| Boot in paywalled WiFi | ✅ | ❌ | ✅ | +| Boot in air-gapped env | ✅ | ❌ | ✅ | +| Complex reasoning | ⚠️ (7B is decent) | ✅ | ✅ (API when available) | +| BSOD analysis | ✅ | ✅ | ✅ | +| SMART disk parsing | ✅ | ✅ | ✅ | +| Driver download | ❌ (needs net) | ✅ | ⚠️ (net required anyway) | +| Backup to cloud | ❌ (needs net) | ✅ | ⚠️ (net required anyway) | + +**Key insight:** The hybrid strategy handles the critical "PC won't boot / broken NIC" scenario that's the most common use case for a rescue USB. Both local and hybrid work offline for diagnostics. The only features that genuinely need a network (driver downloads, cloud backup) require it regardless of LLM strategy. + +--- + +## 8. Model Download & Preparation + +Models are downloaded during the build phase (not at runtime): + +```bash +# Build script: fetch models +mkdir -p build/models/ + +# Primary model — Qwen2.5-7B-Instruct Q4_K_M +wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q4_k_m.gguf \ + -O build/models/qwen2.5-7b-instruct-q4_k_m.gguf + +# Fallback model — Qwen2.5-3B-Instruct Q4_K_M +wget https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.gguf \ + -O build/models/qwen2.5-3b-instruct-q4_k_m.gguf + +# llama-server binary (Linux x86_64 static) +wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server \ + -O build/tools/llama-server +chmod +x build/tools/llama-server +``` + +**Licensing:** Qwen2.5 models use the Apache 2.0 license — free for commercial and personal use. + +--- + +## 9. Implementation Recommendation + +### Phase 1: Prototype (this sprint) + +1. **Download** Qwen2.5-7B-Instruct Q4_K_M (4.4 GB) to the NAS project directory +2. **Build** a minimal Hermes profile ("rescue") with local provider config +3. **Package** llama-server binary +4. **Write** a startup script (launch-llm.sh) that: + - Probes available RAM + - Decides 7B vs 3B model based on free RAM + - Launches llama-server with appropriate flags + - Waits for API readiness + - Launches Hermes agent with local provider config +5. **Test** locally on Shawn's host + +### Phase 2: Integration (post-arch-decision) + +1. Integrate startup script into the bootable USB image +2. Wire model selection into the hardware inventory module (T5) +3. Add API config template for fallback provider + +### Phase 3: Polish + +1. Benchmark model quality on real BSOD dumps +2. Tune prompt templates for diagnostic tasks +3. Optimize context window usage (shorter contexts = faster on local) +4. Consider Knowledge Distillation: fine-tune a smaller model on repair-centric data + +--- + +## 10. Risk Register + +| Risk | Likelihood | Impact | Mitigation | +|------|-----------|--------|------------| +| Target PC has < 8 GB RAM | Medium | High (7B won't load) | Ship 3B fallback model | +| USB 2.0 port (slow loading) | High (old PCs) | Medium (75s load time) | Show progress indicator, use --no-mmap | +| Qwen2.5 underperforms on diagnostics | Low | Medium | Test on real dumps, fall back to API | +| Boot env is WinPE (no llama.cpp) | TBD from T1 | Critical | Pivot to API-only or switch boot env | +| Network available but slow | Medium | Low | Local handles all diagnostics; API only for complex | +| Model licensing changes | Low | Medium | Apache 2.0 is permissive and stable | + +--- + +## Appendix A: Quick-Reference Commands + +```bash +# Start local LLM server +llama-server \ + -m /usb/hermes/models/qwen2.5-7b-instruct-q4_k_m.gguf \ + --host 127.0.0.1 \ + --port 8080 \ + --ctx-size 4096 \ + --n-gpu-layers 0 \ + --no-mmap \ + --flash-attn \ + --threads $(nproc) + +# Verify it's running +curl http://localhost:8080/v1/models + +# Test inference +curl http://localhost:8080/v1/completions \ + -H "Content-Type: application/json" \ + -d '{ + "prompt": "What does BSOD 0x0000001A (MEMORY_MANAGEMENT) indicate?", + "model": "qwen2.5-7b-instruct-q4_k_m", + "max_tokens": 200, + "temperature": 0.1 + }' + +# Hermes provider config (for rescue profile) +# ~/.hermes/profiles/rescue/config.yaml: +# model: +# base_url: http://localhost:8080/v1 +# api_key: "" +# default: qwen2.5-7b-instruct-q4_k_m +``` + +## Appendix B: Automatic Model Selection Script (pseudocode) + +``` +1. Check target PC RAM: free -g | grep Mem +2. If total RAM >= 8 GB → use 7B Q4_K_M (~7 GB needed) +3. If total RAM >= 4 GB → use 3B Q4_K_M (~4 GB needed) +4. If total RAM < 4 GB → fallback mode: API-only IF network, else limited scripted diagnostics +5. Launch llama-server with selected model +6. Wait for healthy endpoint +7. Launch Hermes agent with local provider config +```