canteen-asset-tracker/docs/OCR_PIPELINE.md

OCR / Vision Pipeline

End-to-end: camera capture → text extraction → asset matching → auto-update.

The Canteen Asset Tracker has a multi-stage OCR pipeline that reads machine ID sticker photos, cross-references extracted identifiers against the assets database, and automatically updates matched assets with photo paths and serial numbers.

Pipeline Stages

Camera/Gallery Photo
       │
       ▼
┌──────────────────┐   Stage 1: Vision (Ollama qwen2.5vl:3b)
│  Ollama Vision   │   Runs first — most accurate on complex labels
│  (Python API)    │   Runs on Windows PC (RTX 2080), tunneled via SSH
└────────┬─────────┘    to localhost:11434
         │ (fallback if text < 10 clean chars)
         ▼
┌──────────────────┐   Stage 2: OCR (Tesseract)
│  Tesseract OCR   │   Fallback — works on clean, high-contrast labels
│  (pytesseract)   │   Uses `--psm 6` (uniform block of text)
└────────┬─────────┘
         │
         ▼
┌──────────────────┐   Stage 3: Identifier Extraction
│  Normalize &     │   - Strips label prefixes (S/N:, ID#, Machine ID, etc.)
│  Extract IDs     │   - Removes punctuation, uppercases
└────────┬─────────┘   - Extracts full lines + individual tokens
         │
         ▼
┌──────────────────┐   Stage 4: DB Cross-Reference
│  Match against   │   - Matches normalized IDs against serial_number,
│  assets.db       │     machine_id, connect_id, equipment_id, barcode
└────────┬─────────┘   - Deduplicates by asset id
         │
         ▼
┌──────────────────┐   Stage 5: Auto-Update
│  Write Back to   │   - photo_path → set on matched assets (if blank)
│  Matched Assets  │   - serial_number → extracted from OCR text (if blank)
└──────────────────┘   - GPS coords → set on matched asset (if blank, from EXIF)

API Endpoints

POST /api/ocr — Full OCR pipeline

Accepts an image upload (multipart/form-data). Returns extracted text, matched assets, and auto-updates the database.

Request:

POST /api/ocr
Content-Type: multipart/form-data
file: <image>               # Required: the sticker photo
exif_data: <JSON string>    # Optional: client-side EXIF data for gallery uploads

Response fields:

Field	Description
raw_text	Raw text extracted by Ollama or Tesseract
ocr_source	"ollama", "tesseract", or "none"
machine_id	5-digit machine ID from Connect-ID pattern (XXXXX-XXXXXX)
confidence	"high" (exact pattern match), "low" (loose match), "none"
matched_assets	Array of assets matched via identifier cross-reference
exif_gps	GPS coordinates from photo EXIF (if present)
gps_saved	true if GPS was auto-saved to the matched asset
path	Saved photo URL path (when exif_data provided)
photo_saved	Number of matched assets whose photo_path was auto-updated
serial_saved	Serial number value that was auto-saved to blank-serial matched assets

POST /api/upload/photo — Photo-only upload

Saves a photo without OCR. Returns the saved path and any EXIF GPS data. Use this when the photo was already matched to an asset client-side.

POST /api/match-text — Text matching only

Accepts raw text and finds matching assets. Does not run OCR. Use for barcode scanner input, QR codes, or client-side vision results.

Auto-Update Logic

When a photo is uploaded through /api/ocr and matches database assets:

1. photo_path — If the matched asset has no photo_path set, the saved photo's URL path is written to the asset. This links the photo directly to the asset record for display in the asset detail view.

2. serial_number — If the matched asset has a blank serial_number and the OCR text contains a serial-number pattern (prefixed with S/N, Serial#, Equipment ID, etc.), the extracted serial is written to all matched blank-serial assets.

3. GPS coordinates — If the photo has EXIF GPS data and the matched asset has no lat/lng coordinates, they are auto-saved.

All auto-updates are best-effort and non-critical — they don't fail the OCR endpoint if a DB write errors.

Backfill Script

scripts/backfill_vision_photos.py

Processes all photos in uploads/photos/ that aren't already linked to an asset via photo_path. Useful for:

• Processing photos that were uploaded before the auto-link feature was added
• Retrying photos that failed to match initially
• Bulk-linking a directory of asset photos

# Dry-run: see what would be updated
python3 scripts/backfill_vision_photos.py

# Apply changes
python3 scripts/backfill_vision_photos.py --apply

# Force overwrite existing photo_path values
python3 scripts/backfill_vision_photos.py --apply --force

Options:

Flag	Description
--apply	Write changes to DB (default is dry-run)
--force	Overwrite existing photo_path values on assets
--db	Custom database path
--photos-dir	Custom photos directory

scripts/match_label_photo.py

CLI utility to OCR a single image and display matched assets. Supports --text flag to skip OCR and pass raw text directly.

# OCR a photo
python3 scripts/match_label_photo.py path/to/photo.jpg

# Match against pre-extracted text
python3 scripts/match_label_photo.py --text "S/N: 2500.0100.0025534"

OCR Services

Primary: Ollama Vision (qwen2.5vl:3b)

• Host: Windows gaming PC (RTX 2080), tunneled to localhost:11434
• Accuracy: High — can read text from complex labels, dark backgrounds, angled photos
• Speed: ~5-15 seconds per image (depends on GPU load)
• Endpoint: /api/generate with prompt focused on text extraction

Fallback: Tesseract OCR

• Installation: sudo apt-get install -y tesseract-ocr
• Python: pip install pytesseract Pillow
• Accuracy: Good on clean, high-contrast, well-lit labels
• Mode: --psm 6 (assume uniform block of text)

Availability Check

The server checks both services at startup:

• _HAS_OLLAMA — set if http://127.0.0.1:11434/api/generate responds
• _HAS_TESSERACT — set if pytesseract is importable

Identifier Matching

The normalize_identifier() function in classify_makes.py handles identifier normalization:

1. Strips label prefixes (S/N:, ID#, Machine ID, Monyx ID, etc.)
2. Removes dots, dashes, spaces, slashes, colons
3. Uppercases everything
4. Returns just the alphanumeric core

This normalized string is then searched across serial_number, connect_id, equipment_id, machine_id, and barcode columns in the assets table.

Photo Storage

• Directory: uploads/photos/
• Naming: UUID hex (avoids filename collisions)
• Extensions: .png, .jpg, .jpeg, .dng (configurable via PHOTO_ALLOWED_EXTS)
• Max size: 20 MB (configurable via PHOTO_MAX_SIZE)
• Serving: Mounted at /uploads via FastAPI StaticFiles
• EXIF round-trip: Client reads EXIF before upload, server re-embeds it into saved JPEG