Files

T

shawn 85d460e749 research: LLM strategy deliverable — hybrid local-first recommendation

2026-07-04 03:54:02 -04:00

15 KiB

Raw Blame History

LLM Strategy — Hermes Portable Rescue

Task: T2 — research:llm-strat Author: Ashley (assistant) Date: 2026-07-04 Status: Complete — ready for T3 (workstream:arch-decision)

Executive Summary

Recommended approach: Hybrid Local-First strategy

Layer	Model	When	Storage	RAM
Primary	Qwen2.5-7B-Instruct Q4_K_M (4.4 GB)	Always, offline-first	USB	~7 GB
Lightweight fallback	Qwen2.5-3B-Instruct Q4_K_M (2.0 GB)	Target < 8 GB RAM	USB	~4 GB
API upgrade	Cloud API (OpenCode Go / DeepSeek-V4)	Network available, complex reasoning	—	—

Total USB storage for models: 4.4 GB (primary) + 2.0 GB (fallback) = 6.4 GB Recommended USB drive size: 64 GB+

1. Options Evaluated

1A — Local GGUF Model (Offline-only)

How it works: Ship a pre-quantized GGUF model on the USB drive. On boot, launch llama-server (single binary, ~12 MB statically compiled) which provides an OpenAI-compatible API on localhost. Hermes agent connects to it as its provider.

Best for:

PCs with no working network (broken WiFi driver, dead NIC, paywalled captive portal)
Air-gapped / sensitive environments
Consistent, predictable performance

LLM backpack (llama-server): 12 MB static binary, no Python deps, no shared libraries.

Risks:

Slower than API (10-40 tok/s on CPU vs 50-200+ tok/s via API)
Model size matters — must fit in available RAM after OS overhead
Cannot use large models (70B is out of reach on consumer hardware)

1B — API-based (Cloud-only)

How it works: The rescue agent requires a working internet connection and calls a cloud API (OpenCode Go, OpenRouter, Anthropic, etc.) for every inference.

Best for:

Complex multi-factor reasoning (BSOD chain analysis, rare hardware issues)
Access to state-of-the-art models (Claude, GPT-4, DeepSeek-V4-Pro)
Minimal USB storage overhead (no model files)

Risks:

Crippling failure mode — The most common reason for a rescue USB is a PC with broken network. If the agent can't think without the API, it's useless when you need it most.
Latency on spotty/paywalled connections
Privacy — diagnostic data leaves the target machine
API cost per session

1C — Hybrid (Local-First, API-Enhanced)

How it works: Ship both a local GGUF model and an API config. Default to local inference. If a working internet connection is detected AND the local model's output is low-confidence (or the task is complex enough), fall through to a cloud API call for the specific step.

Best for:

Everything — offline-capable by default, enhanced when possible
Graceful degradation: works on a PC with a fried NIC, gets better on a PC with working WiFi

Risks:

Slightly larger USB image (model files + API config)
Two code paths to test and maintain
Need a heuristic for "when to escalate to API"

2. Model Comparison

Candidate Models for Local Inference

Model	Size (params)	Q4_K_M Size	RAM Needed	Speed (CPU)	Technical Reasoning
Qwen2.5-7B-Instruct	7.6B	4.4 GB	~7 GB	15-25 tok/s	Excellent — strong instruction following
Qwen2.5-Coder-7B-Instruct	7.6B	4.4 GB	~7 GB	15-25 tok/s	Great for code, overfit for general repair
Llama-3.2-3B-Instruct	3.2B	1.9 GB	~4 GB	25-40 tok/s	Good for simple diagnostics, weak at chains
Qwen2.5-3B-Instruct	3.1B	2.0 GB	~4 GB	25-40 tok/s	Good-small model, decent reasoning
Phi-3.5-mini-Instruct	3.8B	2.2 GB	~5 GB	20-35 tok/s	Decent technical, smaller tokenizer
Mistral-7B-v0.3	7.3B	4.4 GB	~7 GB	13-23 tok/s	Strong general, slower than Qwen
Gemma-2-9B	9.2B	5.5 GB	~8 GB	10-18 tok/s	Very strong, but needs more RAM

Recommended: Qwen2.5-7B-Instruct Q4_K_M

Why this model:

Best quality-to-size ratio — Tops the Open LLM Leaderboard in the 7B class, with strong instruction following and reasoning. Perfect for parsing minidump stop codes, SMART attribute values, and multi-step diagnostic chains.
Qwen2.5 tokenizer handles technical English and hex dumps efficiently.
Q4_K_M is the sweet spot — only 1.7% perplexity loss vs FP16, 4x smaller than FP16.
Single file — 4.36 GB for Q4_K_M, no shards to manage.

Why NOT Qwen2.5-Coder-7B:

Overfitted for code generation. A rescue agent needs to reason about stop codes, hardware specs, and repair procedures — not generate code. The general instruct variant handles these better.

Lightweight Fallback: Qwen2.5-3B-Instruct Q4_K_M

1.96 GB on disk, runs in ~4 GB RAM
Covers PCs with only 8 GB total RAM (OS overhead + Hermes leaves ~4-5 GB for the model)
Speedy: 25-40 tok/s on modern CPU

RAM Calculation for Target PCs

Target PC RAM	OS Overhead (Live Linux)	Available for Model	Can Run 7B Q4_K_M?	Can Run 3B Q4_K_M?
4 GB	~1-1.5 GB	~2.5-3 GB	No	Tight
8 GB	~1-1.5 GB	~6.5-7 GB	Yes	Yes
16 GB	~1-1.5 GB	~14.5-15 GB	Yes	Yes
32 GB	~1-1.5 GB	~30 GB	Yes	Yes

Conclusion: The 7B Q4_K_M model requires at least 8 GB of target PC RAM. This covers the vast majority of consumer and gaming PCs from the last 7 years. For low-RAM machines (< 8 GB), fall back to the 3B model.

3. Boot Environment Implications

The LLM strategy is tightly coupled with the boot environment (T1 — research:boot-env).

Boot Env	LLM Viability	Notes
Linux live (Alpine, Arch, custom)	✅ Excellent	Native llama.cpp binary, full RAM access, Python with pip, simple process management
WinPE	❌ Impractical	No native llama.cpp without MSVC runtime. No Python unless embedded. WinPE is heavily stripped (no .NET, no VC++ redist). Running LLM inference requires significant hacks.
Ventoy + Linux ISO	✅ Excellent	Same as Linux live — llama.cpp + Python work natively

The LLM strategy effectively requires a Linux-based boot environment. If the boot-env research (T1) concludes WinPE must be used, this strategy must be revised significantly (likely API-only with offline fallback being extremely limited diagnostic scripts without LLM).

Assuming Linux live environment:

Boot → Start llama-server in background → Start Hermes agent → Hermes connects to localhost:8080

[llama-server]                   [Hermes Agent]
  ┌──────────────┐                ┌──────────────┐
  │ llama-server  │←──localhost──│ Hermes core   │
  │ --port 8080   │    :8080/v1  │ provider:     │
  │ --ctx-size 4096│              │   base_url:   │
  │ model.gguf     │              │   http://     │
  └──────────────┘                │   localhost    │
                                  │   :8080/v1    │
                                  └──────────────┘

4. Hermes Provider Configuration

Hermes supports any OpenAI-compatible API as a provider by setting model.base_url in config.yaml. For the rescue environment, the portable config would be:

# ~/.hermes/config.yaml (rescue profile)
model:
  # If local model is running:
  base_url: http://localhost:8080/v1
  api_key: ""            # or "sk-no-key-required"
  default: qwen2.5-7b-instruct-q4_k_m
  provider: ""           # base_url overrides provider

  # If API fallback (network available):
  # base_url: https://opencode.ai/zen/go/v1/
  # api_key: ${OPENCODE_GO_API_KEY}
  # default: deepseek-v4-flash

Local inference stack:

llama-server — single static binary (~12 MB), started as systemd service or background process
Launched with: llama-server -m /usb/hermes/models/qwen2.5-7b-instruct-q4_k_m.gguf --host 127.0.0.1 --port 8080 --ctx-size 4096 --n-gpu-layers 0 --no-mmap
Hermes agent detects the API is available, sets base_url, and works normally

Key flag: --no-mmap — Prevents memory-mapping the model file, important when running from a USB stick (mmap on slow USB can cause stuttering). Falls back to read-ahead loading.

5. Storage Budget on USB

Component	Size	Notes
Linux live OS + kernel	~500 MB-1.5 GB	Alpine minimal ~200 MB, Arch ~800 MB, custom ~1 GB
llama-server binary	~12 MB	Static build, no deps
Qwen2.5-7B Q4_K_M model	4.4 GB	Primary model
Qwen2.5-3B Q4_K_M model	2.0 GB	Fallback (optional)
Hermes agent (Python + venv)	~200-400 MB	Minimal venv, hermes core
Diagnostic tools	~100-300 MB	smartmontools, dmidecode, stress-ng, MemTest86, dd, etc.
Docker image cache	~0 MB optional	Running native binaries preferred
Total (with both models)	~7.5-9 GB

All models fit comfortably on a 32 GB USB stick. For 64 GB+ sticks, consider adding additional quantizations or a larger model (e.g., Qwen2.5-14B Q4_K_M at ~8 GB for extra reasoning capability).

USB Speed Impact

USB 3.0 (5 Gbps) → ~500 MB/s theoretical
USB 3.1 Gen 2 (10 Gbps) → ~1 GB/s
USB 2.0 (480 Mbps) → ~60 MB/s practical

Model loading time at USB 3.0: ~9 seconds for 4.4 GB Model loading at USB 2.0: ~75 seconds (annoying but acceptable — boot once per session)

Recommendation: Use --no-mmap flag with llama-server to avoid live USB I/O during inference. The model loads into RAM once at startup and operates entirely from RAM afterwards.

6. Speed & Latency Analysis

Scenario	Task	Local 7B Q4_K_M	API (DeepSeek-V4-Flash)
First token	Cold start	~1-3 sec	~0.5-2 sec
BSOD analysis	200 tokens	~10 sec	~3-5 sec
SMART interpretation	400 tokens	~20 sec	~5-8 sec
Repair plan	800 tokens	~35 sec	~8-12 sec

Local inference is 2-5x slower than API but still fast enough for a diagnostic workflow where the user waits for results anyway. BSOD analysis at 10 seconds is perfectly acceptable — the alternative is the user manually Googling stop codes.

7. Offline Capability Assessment

Feature	Local GGUF	API-only	Hybrid
Boot with dead NIC	✅	❌	✅
Boot in paywalled WiFi	✅	❌	✅
Boot in air-gapped env	✅	❌	✅
Complex reasoning	⚠️ (7B is decent)	✅	✅ (API when available)
BSOD analysis	✅	✅	✅
SMART disk parsing	✅	✅	✅
Driver download	❌ (needs net)	✅	⚠️ (net required anyway)
Backup to cloud	❌ (needs net)	✅	⚠️ (net required anyway)

Key insight: The hybrid strategy handles the critical "PC won't boot / broken NIC" scenario that's the most common use case for a rescue USB. Both local and hybrid work offline for diagnostics. The only features that genuinely need a network (driver downloads, cloud backup) require it regardless of LLM strategy.

8. Model Download & Preparation

Models are downloaded during the build phase (not at runtime):

# Build script: fetch models
mkdir -p build/models/

# Primary model — Qwen2.5-7B-Instruct Q4_K_M
wget https://huggingface.co/Qwen/Qwen2.5-7B-Instruct-GGUF/resolve/main/qwen2.5-7b-instruct-q4_k_m.gguf \
  -O build/models/qwen2.5-7b-instruct-q4_k_m.gguf

# Fallback model — Qwen2.5-3B-Instruct Q4_K_M
wget https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.gguf \
  -O build/models/qwen2.5-3b-instruct-q4_k_m.gguf

# llama-server binary (Linux x86_64 static)
wget https://github.com/ggml-org/llama.cpp/releases/latest/download/llama-server \
  -O build/tools/llama-server
chmod +x build/tools/llama-server

Licensing: Qwen2.5 models use the Apache 2.0 license — free for commercial and personal use.

9. Implementation Recommendation

Phase 1: Prototype (this sprint)

Download Qwen2.5-7B-Instruct Q4_K_M (4.4 GB) to the NAS project directory
Build a minimal Hermes profile ("rescue") with local provider config
Package llama-server binary
Write a startup script (launch-llm.sh) that:
- Probes available RAM
- Decides 7B vs 3B model based on free RAM
- Launches llama-server with appropriate flags
- Waits for API readiness
- Launches Hermes agent with local provider config
Test locally on Shawn's host

Phase 2: Integration (post-arch-decision)

Integrate startup script into the bootable USB image
Wire model selection into the hardware inventory module (T5)
Add API config template for fallback provider

Phase 3: Polish

Benchmark model quality on real BSOD dumps
Tune prompt templates for diagnostic tasks
Optimize context window usage (shorter contexts = faster on local)
Consider Knowledge Distillation: fine-tune a smaller model on repair-centric data

10. Risk Register

Risk	Likelihood	Impact	Mitigation
Target PC has < 8 GB RAM	Medium	High (7B won't load)	Ship 3B fallback model
USB 2.0 port (slow loading)	High (old PCs)	Medium (75s load time)	Show progress indicator, use --no-mmap
Qwen2.5 underperforms on diagnostics	Low	Medium	Test on real dumps, fall back to API
Boot env is WinPE (no llama.cpp)	TBD from T1	Critical	Pivot to API-only or switch boot env
Network available but slow	Medium	Low	Local handles all diagnostics; API only for complex
Model licensing changes	Low	Medium	Apache 2.0 is permissive and stable

Appendix A: Quick-Reference Commands

# Start local LLM server
llama-server \
  -m /usb/hermes/models/qwen2.5-7b-instruct-q4_k_m.gguf \
  --host 127.0.0.1 \
  --port 8080 \
  --ctx-size 4096 \
  --n-gpu-layers 0 \
  --no-mmap \
  --flash-attn \
  --threads $(nproc)

# Verify it's running
curl http://localhost:8080/v1/models

# Test inference
curl http://localhost:8080/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "prompt": "What does BSOD 0x0000001A (MEMORY_MANAGEMENT) indicate?",
    "model": "qwen2.5-7b-instruct-q4_k_m",
    "max_tokens": 200,
    "temperature": 0.1
  }'

# Hermes provider config (for rescue profile)
# ~/.hermes/profiles/rescue/config.yaml:
# model:
#   base_url: http://localhost:8080/v1
#   api_key: ""
#   default: qwen2.5-7b-instruct-q4_k_m

Appendix B: Automatic Model Selection Script (pseudocode)

1. Check target PC RAM: free -g | grep Mem
2. If total RAM >= 8 GB → use 7B Q4_K_M (~7 GB needed)
3. If total RAM >= 4 GB → use 3B Q4_K_M (~4 GB needed)
4. If total RAM < 4 GB → fallback mode: API-only IF network, else limited scripted diagnostics
5. Launch llama-server with selected model
6. Wait for healthy endpoint
7. Launch Hermes agent with local provider config

15 KiB Raw Blame History

LLM Strategy — Hermes Portable Rescue

Executive Summary

1. Options Evaluated

1A — Local GGUF Model (Offline-only)

1B — API-based (Cloud-only)

1C — Hybrid (Local-First, API-Enhanced)

2. Model Comparison

Candidate Models for Local Inference

Recommended: Qwen2.5-7B-Instruct Q4_K_M

Lightweight Fallback: Qwen2.5-3B-Instruct Q4_K_M

RAM Calculation for Target PCs

3. Boot Environment Implications

4. Hermes Provider Configuration

5. Storage Budget on USB

USB Speed Impact

6. Speed & Latency Analysis

7. Offline Capability Assessment

8. Model Download & Preparation

9. Implementation Recommendation

Phase 1: Prototype (this sprint)

Phase 2: Integration (post-arch-decision)

Phase 3: Polish

10. Risk Register

Appendix A: Quick-Reference Commands

Appendix B: Automatic Model Selection Script (pseudocode)

15 KiB

Raw Blame History