canteen-asset-tracker/docs/OCR_PIPELINE.md

# OCR / Vision Pipeline

> End-to-end: camera capture → text extraction → asset matching → auto-update.

The Canteen Asset Tracker has a multi-stage OCR pipeline that reads machine ID sticker photos, cross-references extracted identifiers against the assets database, and automatically updates matched assets with photo paths and serial numbers.

## Pipeline Stages

```
Camera/Gallery Photo
       │
       ▼
┌──────────────────┐   Stage 1: Vision (Ollama qwen2.5vl:3b)
│  Ollama Vision   │   Runs first — most accurate on complex labels
│  (Python API)    │   Runs on Windows PC (RTX 2080), tunneled via SSH
└────────┬─────────┘    to localhost:11434
         │ (fallback if text < 10 clean chars)
         ▼
┌──────────────────┐   Stage 2: OCR (Tesseract)
│  Tesseract OCR   │   Fallback — works on clean, high-contrast labels
│  (pytesseract)   │   Uses `--psm 6` (uniform block of text)
└────────┬─────────┘
         │
         ▼
┌──────────────────┐   Stage 3: Identifier Extraction
│  Normalize &     │   - Strips label prefixes (S/N:, ID#, Machine ID, etc.)
│  Extract IDs     │   - Removes punctuation, uppercases
└────────┬─────────┘   - Extracts full lines + individual tokens
         │
         ▼
┌──────────────────┐   Stage 4: DB Cross-Reference
│  Match against   │   - Matches normalized IDs against serial_number,
│  assets.db       │     machine_id, connect_id, equipment_id, barcode
└────────┬─────────┘   - Deduplicates by asset id
         │
         ▼
┌──────────────────┐   Stage 5: Auto-Update
│  Write Back to   │   - photo_path → set on matched assets (if blank)
│  Matched Assets  │   - serial_number → extracted from OCR text (if blank)
└──────────────────┘   - GPS coords → set on matched asset (if blank, from EXIF)
```

## API Endpoints

### POST `/api/ocr` — Full OCR pipeline

Accepts an image upload (multipart/form-data). Returns extracted text, matched assets, and auto-updates the database.

**Request:**
```
POST /api/ocr
Content-Type: multipart/form-data
file: <image>               # Required: the sticker photo
exif_data: <JSON string>    # Optional: client-side EXIF data for gallery uploads
```

**Response fields:**
| Field | Description |
|-------|-------------|
| `raw_text` | Raw text extracted by Ollama or Tesseract |
| `ocr_source` | `"ollama"`, `"tesseract"`, or `"none"` |
| `machine_id` | 5-digit machine ID from Connect-ID pattern (XXXXX-XXXXXX) |
| `confidence` | `"high"` (exact pattern match), `"low"` (loose match), `"none"` |
| `matched_assets` | Array of assets matched via identifier cross-reference |
| `exif_gps` | GPS coordinates from photo EXIF (if present) |
| `gps_saved` | `true` if GPS was auto-saved to the matched asset |
| `path` | Saved photo URL path (when `exif_data` provided) |
| `photo_saved` | Number of matched assets whose `photo_path` was auto-updated |
| `serial_saved` | Serial number value that was auto-saved to blank-serial matched assets |

### POST `/api/upload/photo` — Photo-only upload

Saves a photo without OCR. Returns the saved path and any EXIF GPS data.
Use this when the photo was already matched to an asset client-side.

### POST `/api/match-text` — Text matching only

Accepts raw text and finds matching assets. Does not run OCR.
Use for barcode scanner input, QR codes, or client-side vision results.

## Auto-Update Logic

When a photo is uploaded through `/api/ocr` and matches database assets:

1. **photo_path** — If the matched asset has no `photo_path` set, the saved photo's URL path is written to the asset. This links the photo directly to the asset record for display in the asset detail view.

2. **serial_number** — If the matched asset has a blank `serial_number` and the OCR text contains a serial-number pattern (prefixed with S/N, Serial#, Equipment ID, etc.), the extracted serial is written to all matched blank-serial assets.

3. **GPS coordinates** — If the photo has EXIF GPS data and the matched asset has no lat/lng coordinates, they are auto-saved.

All auto-updates are best-effort and non-critical — they don't fail the OCR endpoint if a DB write errors.

## Backfill Script

### `scripts/backfill_vision_photos.py`

Processes all photos in `uploads/photos/` that aren't already linked to an asset via `photo_path`. Useful for:

- Processing photos that were uploaded before the auto-link feature was added
- Retrying photos that failed to match initially
- Bulk-linking a directory of asset photos

```bash
# Dry-run: see what would be updated
python3 scripts/backfill_vision_photos.py

# Apply changes
python3 scripts/backfill_vision_photos.py --apply

# Force overwrite existing photo_path values
python3 scripts/backfill_vision_photos.py --apply --force
```

Options:
| Flag | Description |
|------|-------------|
| `--apply` | Write changes to DB (default is dry-run) |
| `--force` | Overwrite existing `photo_path` values on assets |
| `--db` | Custom database path |
| `--photos-dir` | Custom photos directory |

## `scripts/match_label_photo.py`

CLI utility to OCR a single image and display matched assets. Supports `--text` flag to skip OCR and pass raw text directly.

```bash
# OCR a photo
python3 scripts/match_label_photo.py path/to/photo.jpg

# Match against pre-extracted text
python3 scripts/match_label_photo.py --text "S/N: 2500.0100.0025534"
```

## OCR Services

### Primary: Ollama Vision (qwen2.5vl:3b)

- **Host:** Windows gaming PC (RTX 2080), tunneled to `localhost:11434`
- **Accuracy:** High — can read text from complex labels, dark backgrounds, angled photos
- **Speed:** ~5-15 seconds per image (depends on GPU load)
- **Endpoint:** `/api/generate` with prompt focused on text extraction

### Fallback: Tesseract OCR

- **Installation:** `sudo apt-get install -y tesseract-ocr`
- **Python:** `pip install pytesseract Pillow`
- **Accuracy:** Good on clean, high-contrast, well-lit labels
- **Mode:** `--psm 6` (assume uniform block of text)

### Availability Check

The server checks both services at startup:
- `_HAS_OLLAMA` — set if `http://127.0.0.1:11434/api/generate` responds
- `_HAS_TESSERACT` — set if `pytesseract` is importable

## Identifier Matching

The `normalize_identifier()` function in `classify_makes.py` handles identifier normalization:

1. Strips label prefixes (`S/N:`, `ID#`, `Machine ID`, `Monyx ID`, etc.)
2. Removes dots, dashes, spaces, slashes, colons
3. Uppercases everything
4. Returns just the alphanumeric core

This normalized string is then searched across `serial_number`, `connect_id`, `equipment_id`, `machine_id`, and `barcode` columns in the assets table.

## Photo Storage

- **Directory:** `uploads/photos/`
- **Naming:** UUID hex (avoids filename collisions)
- **Extensions:** `.png`, `.jpg`, `.jpeg`, `.dng` (configurable via `PHOTO_ALLOWED_EXTS`)
- **Max size:** 20 MB (configurable via `PHOTO_MAX_SIZE`)
- **Serving:** Mounted at `/uploads` via FastAPI StaticFiles
- **EXIF round-trip:** Client reads EXIF before upload, server re-embeds it into saved JPEG