# LLM Strategy — Hermes Portable Rescue

**Task:** T2 — research:llm-strat
**Author:** Ashley (assistant)
**Date:** 2026-07-04
**Status:** Complete — ready for T3 (workstream:arch-decision)

---

## Executive Summary

**Recommended approach: Hybrid Local-First strategy**

| Layer | Model | When | Storage | RAM |
|-------|-------|------|---------|-----|
| **Primary** | Qwen2.5-7B-Instruct Q4_K_M (4.4 GB) | Always, offline-first | USB | ~7 GB |
| **Lightweight fallback** | Qwen2.5-3B-Instruct Q4_K_M (2.0 GB) | Target < 8 GB RAM | USB | ~4 GB |
| **API upgrade** | Cloud API (OpenCode Go / DeepSeek-V4) | Network available, complex reasoning | — | — |

**Total USB storage for models:** 4.4 GB (primary) + 2.0 GB (fallback) = 6.4 GB
**Recommended USB drive size:** 64 GB+

---

## 1. Options Evaluated

### 1A — Local GGUF Model (Offline-only)

**How it works:** Ship a pre-quantized GGUF model on the USB drive. On boot, launch `llama-server` (single binary, ~12 MB statically compiled) which provides an OpenAI-compatible API on localhost. Hermes agent connects to it as its provider.

**Best for:**
- PCs with no working network (broken WiFi driver, dead NIC, paywalled captive portal)
- Air-gapped / sensitive environments
- Consistent, predictable performance

**LLM backpack (llama-server):** 12 MB static binary, no Python deps, no shared libraries.

**Risks:**
- Slower than API (10-40 tok/s on CPU vs 50-200+ tok/s via API)
- Model size matters — must fit in available RAM after OS overhead
- Cannot use large models (70B is out of reach on consumer hardware)

### 1B — API-based (Cloud-only)

**How it works:** The rescue agent requires a working internet connection and calls a cloud API (OpenCode Go, OpenRouter, Anthropic, etc.) for every inference.

**Best for:**
- Complex multi-factor reasoning (BSOD chain analysis, rare hardware issues)
- Access to state-of-the-art models (Claude, GPT-4, DeepSeek-V4-Pro)
- Minimal USB storage overhead (no model files)

**Risks:**
- **Crippling failure mode** — The most common reason for a rescue USB is a PC with broken network. If the agent can't think without the API, it's useless when you need it most.
- Latency on spotty/paywalled connections
- Privacy — diagnostic data leaves the target machine
- API cost per session

### 1C — Hybrid (Local-First, API-Enhanced)

**How it works:** Ship both a local GGUF model and an API config. Default to local inference. If a working internet connection is detected AND the local model's output is low-confidence (or the task is complex enough), fall through to a cloud API call for the specific step.

**Best for:**
- Everything — offline-capable by default, enhanced when possible
- Graceful degradation: works on a PC with a fried NIC, gets *better* on a PC with working WiFi

**Risks:**
- Slightly larger USB image (model files + API config)
- Two code paths to test and maintain
- Need a heuristic for "when to escalate to API"

---

## 2. Model Comparison

### Candidate Models for Local Inference

| Model | Size (params) | Q4_K_M Size | RAM Needed | Speed (CPU) | Technical Reasoning |
|-------|--------------|-------------|------------|-------------|-------------------|
| **Qwen2.5-7B-Instruct** | 7.6B | **4.4 GB** | ~7 GB | 15-25 tok/s | Excellent — strong instruction following |
| Qwen2.5-Coder-7B-Instruct | 7.6B | 4.4 GB | ~7 GB | 15-25 tok/s | Great for code, overfit for general repair |
| Llama-3.2-3B-Instruct | 3.2B | **1.9 GB** | ~4 GB | 25-40 tok/s | Good for simple diagnostics, weak at chains |
| Qwen2.5-3B-Instruct | 3.1B | **2.0 GB** | ~4 GB | 25-40 tok/s | Good-small model, decent reasoning |
| Phi-3.5-mini-Instruct | 3.8B | 2.2 GB | ~5 GB | 20-35 tok/s | Decent technical, smaller tokenizer |
| Mistral-7B-v0.3 | 7.3B | 4.4 GB | ~7 GB | 13-23 tok/s | Strong general, slower than Qwen |
| Gemma-2-9B | 9.2B | 5.5 GB | ~8 GB | 10-18 tok/s | Very strong, but needs more RAM |

### Recommended: Qwen2.5-7B-Instruct Q4_K_M

**Why this model:**
1. **Best quality-to-size ratio** — Tops the Open LLM Leaderboard in the 7B class, with strong instruction following and reasoning. Perfect for parsing minidump stop codes, SMART attribute values, and multi-step diagnostic chains.
2. **Qwen2.5 tokenizer** handles technical English and hex dumps efficiently.
3. **Q4_K_M** is the sweet spot — only 1.7% perplexity loss vs FP16, 4x smaller than FP16.
4. **Single file** — 4.36 GB for Q4_K_M, no shards to manage.

**Why NOT Qwen2.5-Coder-7B:**
- Overfitted for code generation. A rescue agent needs to reason about stop codes, hardware specs, and repair procedures — not generate code. The general instruct variant handles these better.

### Lightweight Fallback: Qwen2.5-3B-Instruct Q4_K_M

- 1.96 GB on disk, runs in ~4 GB RAM
- Covers PCs with only 8 GB total RAM (OS overhead + Hermes leaves ~4-5 GB for the model)
- Speedy: 25-40 tok/s on modern CPU

### RAM Calculation for Target PCs

| Target PC RAM | OS Overhead (Live Linux) | Available for Model | Can Run 7B Q4_K_M? | Can Run 3B Q4_K_M? |
|--------------|--------------------------|-------------------|--------------------|--------------------|
| 4 GB | ~1-1.5 GB | ~2.5-3 GB | No | **Tight** |
| 8 GB | ~1-1.5 GB | ~6.5-7 GB | **Yes** | Yes |
| 16 GB | ~1-1.5 GB | ~14.5-15 GB | Yes | Yes |
| 32 GB | ~1-1.5 GB | ~30 GB | Yes | Yes |

**Conclusion:** The 7B Q4_K_M model requires at least 8 GB of target PC RAM. This covers the vast majority of consumer and gaming PCs from the last 7 years. For low-RAM machines (< 8 GB), fall back to the 3B model.

---

## 3. Boot Environment Implications

The LLM strategy is **tightly coupled** with the boot environment (T1 — research:boot-env).

| Boot Env | LLM Viability | Notes |
|----------|--------------|-------|
| **Linux live** (Alpine, Arch, custom) | ✅ **Excellent** | Native llama.cpp binary, full RAM access, Python with pip, simple process management |
| **WinPE** | ❌ **Impractical** | No native llama.cpp without MSVC runtime. No Python unless embedded. WinPE is heavily stripped (no .NET, no VC++ redist). Running LLM inference requires significant hacks. |
| **Ventoy + Linux ISO** | ✅ **Excellent** | Same as Linux live — llama.cpp + Python work natively |

**The LLM strategy effectively requires a Linux-based boot environment.** If the boot-env research (T1) concludes WinPE must be used, this strategy must be revised significantly (likely API-only with offline fallback being extremely limited diagnostic scripts without LLM).

**Assuming Linux live environment:**

```
Boot → Start llama-server in background → Start Hermes agent → Hermes connects to localhost:8080

[llama-server]                   [Hermes Agent]
  ┌──────────────┐                ┌──────────────┐
  │ llama-server  │←──localhost──│ Hermes core   │
  │ --port 8080   │    :8080/v1  │ provider:     │
  │ --ctx-size 4096│              │   base_url:   │
  │ model.gguf     │              │   http://     │
  └──────────────┘                │   localhost    │
                                  │   :8080/v1    │
                                  └──────────────┘
```

---

## 4. Hermes Provider Configuration

Hermes supports **any OpenAI-compatible API** as a provider by setting `model.base_url` in config.yaml. For the rescue environment, the portable config would be:

```yaml
# ~/.hermes/config.yaml (rescue profile)
model:
  # If local model is running:
  base_url: http://localhost:8080/v1
  api_key: ""            # or "sk-no-key-required"
  default: qwen2.5-7b-instruct-q4_k_m
  provider: ""           # base_url overrides provider

  # If API fallback (network available):
  # base_url: https://opencode.ai/zen/go/v1/
  # api_key: ${OPENCODE_GO_API_KEY}
  # default: deepseek-v4-flash
```

**Local inference stack:**
1. `llama-server` — single static binary (~12 MB), started as systemd service or background process
2. Launched with: `llama-server -m /usb/hermes/models/qwen2.5-7b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx-size 4096 --n-gpu-layers 0 --no-mmap`
3. Hermes agent detects the API is available, sets base_url, and works normally

**Key flag: `--no-mmap`** — Prevents memory-mapping the model file, important when running from a USB stick (mmap on slow USB can cause stuttering). Falls back to read-ahead loading.

---

## 5. Storage Budget on USB

| Component | Size | Notes |
|-----------|------|-------|
| Linux live OS + kernel | ~500 MB-1.5 GB | Alpine minimal ~200 MB, Arch ~800 MB, custom ~1 GB |
| llama-server binary | ~12 MB | Static build, no deps |
| Qwen2.5-7B Q4_K_M model | **4.4 GB** | Primary model |
| Qwen2.5-3B Q4_K_M model | **2.0 GB** | Fallback (optional) |
| Hermes agent (Python + venv) | ~200-400 MB | Minimal venv, hermes core |
| Diagnostic tools | ~100-300 MB | smartmontools, dmidecode, stress-ng, MemTest86, dd, etc. |
| Docker image cache | ~0 MB optional | Running native binaries preferred |
| **Total (with both models)** | **~7.5-9 GB** | |

**All models fit comfortably on a 32 GB USB stick.** For 64 GB+ sticks, consider adding additional quantizations or a larger model (e.g., Qwen2.5-14B Q4_K_M at ~8 GB for extra reasoning capability).

### USB Speed Impact

- USB 3.0 (5 Gbps) → ~500 MB/s theoretical
- USB 3.1 Gen 2 (10 Gbps) → ~1 GB/s
- USB 2.0 (480 Mbps) → ~60 MB/s practical

Model loading time at USB 3.0: ~9 seconds for 4.4 GB
Model loading at USB 2.0: ~75 seconds (annoying but acceptable — boot once per session)

**Recommendation:** Use `--no-mmap` flag with llama-server to avoid live USB I/O during inference. The model loads into RAM once at startup and operates entirely from RAM afterwards.

---

## 6. Speed & Latency Analysis

| Scenario | Task | Local 7B Q4_K_M | API (DeepSeek-V4-Flash) |
|----------|------|-----------------|------------------------|
| First token | Cold start | ~1-3 sec | ~0.5-2 sec |
| BSOD analysis | 200 tokens | ~10 sec | ~3-5 sec |
| SMART interpretation | 400 tokens | ~20 sec | ~5-8 sec |
| Repair plan | 800 tokens | ~35 sec | ~8-12 sec |

**Local inference is 2-5x slower than API** but still fast enough for a diagnostic workflow where the user waits for results anyway. BSOD analysis at 10 seconds is perfectly acceptable — the alternative is the user manually Googling stop codes.

---

## 7. Offline Capability Assessment

| Feature | Local GGUF | API-only | Hybrid |
|---------|-----------|----------|--------|
| Boot with dead NIC | ✅ | ❌ | ✅ |
| Boot in paywalled WiFi | ✅ | ❌ | ✅ |
| Boot in air-gapped env | ✅ | ❌ | ✅ |
| Complex reasoning | ⚠️ (7B is decent) | ✅ | ✅ (API when available) |
| BSOD analysis | ✅ | ✅ | ✅ |
| SMART disk parsing | ✅ | ✅ | ✅ |
| Driver download | ❌ (needs net) | ✅ | ⚠️ (net required anyway) |
| Backup to cloud | ❌ (needs net) | ✅ | ⚠️ (net required anyway) |

**Key insight:** The hybrid strategy handles the critical "PC won't boot / broken NIC" scenario that's the most common use case for a rescue USB. Both local and hybrid work offline for diagnostics. The only features that genuinely need a network (driver downloads, cloud backup) require it regardless of LLM strategy.

---

## 8. Model Download & Preparation

Models are downloaded during the build phase (not at runtime):

```bash
# Build script: fetch models
mkdir -p build/models/

# Primary model — Qwen2.5-7B-Instruct Q4_K_M
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q4_k_m.gguf \
  -O build/models/qwen2.5-7b-instruct-q4_k_m.gguf

# Fallback model — Qwen2.5-3B-Instruct Q4_K_M
wget https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.gguf \
  -O build/models/qwen2.5-3b-instruct-q4_k_m.gguf

# llama-server binary (Linux x86_64 static)
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server \
  -O build/tools/llama-server
chmod +x build/tools/llama-server
```

**Licensing:** Qwen2.5 models use the Apache 2.0 license — free for commercial and personal use.

---

## 9. Implementation Recommendation

### Phase 1: Prototype (this sprint)

1. **Download** Qwen2.5-7B-Instruct Q4_K_M (4.4 GB) to the NAS project directory
2. **Build** a minimal Hermes profile ("rescue") with local provider config
3. **Package** llama-server binary
4. **Write** a startup script (launch-llm.sh) that:
   - Probes available RAM
   - Decides 7B vs 3B model based on free RAM
   - Launches llama-server with appropriate flags
   - Waits for API readiness
   - Launches Hermes agent with local provider config
5. **Test** locally on Shawn's host

### Phase 2: Integration (post-arch-decision)

1. Integrate startup script into the bootable USB image
2. Wire model selection into the hardware inventory module (T5)
3. Add API config template for fallback provider

### Phase 3: Polish

1. Benchmark model quality on real BSOD dumps
2. Tune prompt templates for diagnostic tasks
3. Optimize context window usage (shorter contexts = faster on local)
4. Consider Knowledge Distillation: fine-tune a smaller model on repair-centric data

---

## 10. Risk Register

| Risk | Likelihood | Impact | Mitigation |
|------|-----------|--------|------------|
| Target PC has < 8 GB RAM | Medium | High (7B won't load) | Ship 3B fallback model |
| USB 2.0 port (slow loading) | High (old PCs) | Medium (75s load time) | Show progress indicator, use --no-mmap |
| Qwen2.5 underperforms on diagnostics | Low | Medium | Test on real dumps, fall back to API |
| Boot env is WinPE (no llama.cpp) | TBD from T1 | Critical | Pivot to API-only or switch boot env |
| Network available but slow | Medium | Low | Local handles all diagnostics; API only for complex |
| Model licensing changes | Low | Medium | Apache 2.0 is permissive and stable |

---

## Appendix A: Quick-Reference Commands

```bash
# Start local LLM server
llama-server \
  -m /usb/hermes/models/qwen2.5-7b-instruct-q4_k_m.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 4096 \
  --n-gpu-layers 0 \
  --no-mmap \
  --flash-attn \
  --threads $(nproc)

# Verify it's running
curl http://localhost:8080/v1/models

# Test inference
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What does BSOD 0x0000001A (MEMORY_MANAGEMENT) indicate?",
    "model": "qwen2.5-7b-instruct-q4_k_m",
    "max_tokens": 200,
    "temperature": 0.1
  }'

# Hermes provider config (for rescue profile)
# ~/.hermes/profiles/rescue/config.yaml:
# model:
#   base_url: http://localhost:8080/v1
#   api_key: ""
#   default: qwen2.5-7b-instruct-q4_k_m
```

## Appendix B: Automatic Model Selection Script (pseudocode)

```
1. Check target PC RAM: free -g | grep Mem
2. If total RAM >= 8 GB → use 7B Q4_K_M (~7 GB needed)
3. If total RAM >= 4 GB → use 3B Q4_K_M (~4 GB needed)
4. If total RAM < 4 GB → fallback mode: API-only IF network, else limited scripted diagnostics
5. Launch llama-server with selected model
6. Wait for healthy endpoint
7. Launch Hermes agent with local provider config
```