Files
canteen-seed-import/seed-data/reference_handbook_filled.md
T
shawn 457e7794a0 🌱 Complete seed import tool — all 17 kanban tasks done
Modules:
- parser.py: Full Excel parser (1,848 machines, Disney mapping, Pattern A+B parsing)
- db_writer.py: SQLite writer with per-field update policy (seed + update modes)
- backup.py: GPS/photo backup, restore, and compare
- reporter.py: Comprehensive validation report generator
- main.py: CLI entry point

Database: assets.db with 1,848 machines across 36 columns
Handbook: reference_handbook.md v2.0 — all sections finalized
Report: validation_report.md — 184 OCR corrections, 143 unknown machines
2026-05-24 21:50:46 -04:00

615 lines
19 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# reference_handbook
# 🌱 Seed Data Import — Reference Handbook
> This handbook documents all extraction rules, logic, and reference data
for the canteen seed data import pipeline.
>
>
> Sections marked **FILL_IN** need your domain expertise.
> Sections marked **✅ READY** Ive pre-filled from data analysis.
>
---
## 1. Customer Column → Company/Domain
**Status: ✅ READY**
The `Customer` column in the seed data maps directly to the `company` field in the database.
Its already clean — just the business name. No parsing needed.
### Disney Customers (prefix `D-`)
Customers starting with `D-` are Disney properties. These map to `disney_park`:
| Customer prefix / pattern | Disney Park | Category |
| --- | --- | --- |
| `D-Magic Kingdom` | `magic-kingdom` | Park |
| `D-Epcot` | `epcot` | Park |
| `D-Hollywood Studios` | `hollywood-studios` | Park |
| `D-Animal Kingdom` | `animal-kingdom` | Park |
| `D-Disney Springs` | `disney-springs` | Park |
| `D-ContemporaryHotel` | `resort` | Resort |
| `D-POLYNESIAN RESORT` | `resort` | Resort |
| `D-Port Orleans` | `resort` | Resort |
| `D-CORONADO SPRINGS` | `resort` | Resort |
| `D-ART OF ANIMATION` | `resort` | Resort |
| `D-POP CENTURY` | `resort` | Resort |
| `D-All Star ...` | `resort` | Resort |
| `D-DISNEY WORLD SS` | `office` | Support services |
**FILL_IN**: Add any Disney customer patterns I missed:
-
-
---
## 2. Place Column → Location Fields
**Status: ✅ READY** (patterns identified)
The `Place` column has **two main patterns** — the extraction engine must detect which pattern applies and parse accordingly.
### Pattern A: Simple — ~90% of rows
**Format:** `Venue Name-Venue Detail` (dash-separated, repetitive)
**Examples:**
```
Sygma - Vending-Sygma - Vending Breakroom
Imperial Dade-Imperial Dade- Breakroom
Ancora Apt.-Ancora Apt. Breakroom
Delamarre-Delamarre Breakroom
BMW Service-BMW Breakroom
Kisselback Ford-Kisselback Ford Breakroom
Sam's Club-Sam's Club
BREAK AREA-BREAK AREA
Building 1180-Building 1180
```
**Rule:** Take the last meaningful segment (after the last dash). Strip:
- `Breakroom`, `Br`, `Break Room`, `BREAKROOM`, `BREAK AREA`
- `Vending Area`, `Vending`
- Trailing whitespace and punctuation
### Pattern B: Complex Disney — ~10% of rows
**Format:** `PersonName - Address-G-Zone FLOOR`
**Examples:**
```
Todd J - 901 Tinberline Dr-G-REAR 6TH FL
Carrianne C - 1251 Riversida Dr-G-PO RIVERSIDE 80S
Justin M - 1536 Buena Vista-Team Disney North Win
Penny D - 3520 Ft Wilderness-G-BUS STOP
Sarah W - 4401 Floridian Way-G-BUILDING 9 LOBBY
Brenda G - 2101 Epcot Resorts-Hotel by Room 1290
Jeremy B - 1960 Broadway-2nd FL Brkrm Vacat Club D
```
**Extraction steps:**
1. Strip person name prefix: `<FirstName> <Initial>` or `<FirstName> <LastInit> -`
2. Strip address segment (number + street name)
3. Extract floor info: `NTH FL`, `Nth floor`, `FL N`, `1ST FL`, etc.
1. Extract Building info: `B N`, `Bldg N`, `Mermaid N`, etc
2. Whatever remains is the clean `place` name
### Edge Cases
```
FL8108-Vending Area → place="FL8108", no venue
1st FL Wing-1st Fl Breakroom → floor="1st Fl", place="Wing"
Epcot-Imagination → place="Imagination"
C-MK EASTGATE SECURITY-Be Our Guest → place="Be Our Guest"
Southern Tech University:Sun Life-Sun Life → place="Sun Life" (remove university prefix)
```
### Floor Extraction Patterns
| Pattern | Example | Normalized |
| --- | --- | --- |
| `NTH FL` | `6TH FL` | `6th Floor` |
| `Nth floor` | `7th floor` | `7th Floor` |
| `FL N` | `FL 2` | `Floor 2` |
| `NST FL` | `1ST FL` | `1st Floor` |
**FILL_IN**: Add any special Place patterns or corrections you know about:
-
-
-
---
## 3. GPS Derivation for Multi-Floor Machines
**Status: ⚠️ NEEDS FILL_IN**
### Base GPS Strategy
For machines with no GPS data, we need to derive coordinates:
1. Use existing GPS from other machines at the same address
### Floor Offset Rules
When a base GPS coordinate is found for the building, adjust for floor:
**FILL_IN**: What offset values should we use per floor?
- Does the lat/lng change measurably per floor? No, Machines are usually in the same area directly above. Usually, not always
### GPS Priority
When multiple machines exist at the same address:
1. Use existing GPS from the DB (field-collected is authoritative)
2. Look for other machines on the same floor, on floors above or below, and in the same building
3. Never overwrite existing GPS (preserve policy)
---
## 4. Serial Number Validation & OCR Correction
**Status: ✅ READY** (data analyzed)
### Standard Serial Formats
| Format | Count | Example | Notes |
| --- | --- | --- | --- |
| All digits (10-12 chars) | 1,220 | `168020939`, `471011958` | Crane standard format |
| Starts with `1` (10+ digits) | 175 | `1-16018993`, `112034419` | AMS/Crane prefixed |
| Starts with `222` | various | `222001280013` | Newer Crane format |
| Crane model-prefixed | various | `472-052119`, `449-011276` | Model + serial |
| All digits shorter | various | `124473010109` | USI/Mercato format |
### Likely OCR Errors Found
| Machine ID | Raw Serial | Suspected Correct | Pattern |
| --- | --- | --- | --- |
| 60017 | `15S491811573` | `155491811573` | `S``5` |
| 60025 | `15S491811581` | `155491811581` | `S``5` |
| 60028 | `I5s331807595` | `155331807595` | `I``1`, `s``5` |
| 60032 | `I5SS331807593` | `1555331807593` | `I5SS``1555` |
### Known Character Confusions
| Mistyped | Should Be | Context |
| --- | --- | --- |
| `S` | `5` | In numeric serials — S looks like 5 in some fonts |
| `s` | `5` | Same — lowercase s scanned as 5 |
| `I` | `1` | Capital I looks like 1 |
| `O` | `0` | Letter O vs zero |
| `B` | `8` | B scanned as 8 (less common) |
| `G` | `6` | G scanned as 6 (less common) |
**FILL_IN**: Add any other character confusions youve seen:
- ___________ → ___________
- ___________ → ___________
### Placeholder / Invalid Serials
| Machine ID | Serial | Action |
| --- | --- | --- |
| 60031 | `520` | Flag — likely incomplete or placeholder |
| 68194 | `123` | Flag — clear placeholder |
| 99660 | `00` | Flag — clear placeholder |
**FILL_IN**: How should we handle flagged serials? clear it.
---
### Unknown Machine Identification by Serial
**FILL_IN**: How can we identify an unknown machine from its serial number?
- What do different serial prefixes mean? We can look at known good serials and compare with makes/models to determine structure
- Can we cross-reference with Make/Model/Type? We look at known good machines, and look at their serials to figure out structure
- Any known serial → machine type mappings?
---
---
## 5. Type/Class/Make/Model Consolidation
**Status: ✅ READY** (data analyzed)
### The Priority Rule
The `Type` column is the **best source** for Class/Make/Model. Parse it first,
then fill gaps from the standalone columns.
### Type Column Format
> `CLASS (MAKE MODEL)` — e.g., `Snack (AMS Sensit 3)`
>
**Extraction:**
```
"Snack (AMS Sensit 3)"
→ Class = "Snack"
→ Make = "AMS"
→ Model = "Sensit 3"
"GF Food (Crane 472)"
→ Class = "Food"
→ Make = "Crane"
→ Model = "472"
"Bev (Royal GIII)"
→ Class = "Bev"
→ Make = "Royal"
→ Model = "GIII"
"Unknown"
→ All three = "Unknown" (leave as-is)
```
Make sure that if the Class, make, model, etc has a GF prefix then strip it.
### Type Patterns Found
| Type Pattern | Count | Class | Make | Model |
| --- | --- | --- | --- | --- |
| `Bev (Royal GIII)` | 299 | Bev | Royal | GIII |
| `Snack (Crane Merchant Media)` | 287 | Snack | Crane | Merchant Media |
| `Bev (Vendo 621/721/821)` | 235 | Bev | Vendo | 621/721/821 |
| `GF Food (Crane 472)` | 151 | Food | Crane | 472 |
| `Snack (Crane 15x/16x)` | 76 | Snack | Crane | 15x/16x |
| `Unknown` | 67 | Unknown | Unknown | Unknown |
| `GF Bev (DN 200E)` | 49 | Bev | DN | 200E |
| `GF Bev (DN 5800)` | 35 | GF Bev | DN | 5800 |
| `Snack (Crane 186)` | 34 | Snack | Crane | 186 |
| `GF Bev (DN BevMax 4)` | 34 | GF Bev | DN | BevMax 4 |
| `Bev (DN 501E/600E/276E)` | 34 | Bev | DN | 501E/600E/276E |
| `Snack (Crane 472)` | 26 | Snack | Crane | 472 |
| `Bev (DN 276E)` | 25 | Bev | DN | 276E |
| `GF Bev (DN 3800 BevMax 4)` | 20 | GF Bev | DN | 3800 BevMax 4 |
| `GF Bev (DN Baby BevMax)` | 19 | GF Bev | DN | Baby BevMax |
| `Bev (DN 501E)` | 18 | Bev | DN | 501E |
| `GF Bev (DN BevMax)` | 18 | GF Bev | DN | BevMax |
| `Snack/Bev (Crane 472)` | 16 | Snack/Bev | Crane | 472 |
| `Snack (Crane 187)` | 16 | Snack | Crane | 187 |
| `NATIONAL - 168 SERIES` | 16 | Snack | Crane | 168 |
| `VENDO - 721` | 14 | Bev | Vendo | 721 |
| `Snack (AMS 3561)` | 12 | Snack | AMS | 3561 |
| `Snack (USI Mercato)` | 10 | Snack | USI | Mercato |
| `Snack (VE)` | 10 | Snack | VE | Unknown |
| `NATIONAL - 186` | 10 | Snack | Crane | 186 |
| `NATIONAL - 187` | 10 | Snack | Crane | 187 |
### Special Parse Cases
| Raw Type | Rule |
| --- | --- |
| `NATIONAL - 168 SERIES` | Class=Snack, Make=Crane (National=Crane brand), Model=168 |
| `VENDO - 721` | Class=Bev (if Type doesnt have Class), Make=Vendo, Model=721 |
| `Snack` (no parens) | Class=Snack, check Make/Model columns for fill |
| `Bev` (no parens) | Class=Bev, check Make/Model columns |
| `Unknown` | Leave all Unknown |
### Filling Unknowns
After parsing Type, check the standalone `Make` and `Model` columns to fill gaps.
If Type-based Make is empty but standalone Make has a value, use it.
If both are empty/Unknown, flag for review.
**FILL_IN**: Are there any Type→Make/Model mappings Ive missed?
-
-
### Make Normalization
**FILL_IN**: Consolidate these make variants:
| Raw Value | Normalized To |
| --- | --- |
| `Crane`, `Crane Co`, `Crane Nat`, `NATIONAL` | `Crane` |
| `DN`, `Dixie Narco` | `DN` |
| `Royal`, `Royal Vendors` | `Royal` |
| `Vendo`, `Vendo Co` | `Vendo` |
| `USI`, `U Select It` | `USI` |
| `AMS`, `Automatic Merchandising` | `AMS` |
| `VE`, `Vending Equipment` | ___________ |
| `AP`, `Automatic Products` | ___________ |
| `Faz`, `Fas` | ___________ |
### Class Normalization
| Raw Value | Normalized To |
| --- | --- |
| `Snack`, `Snack/Food` | `Snack` |
| `Bev` | `Bev` |
| `GF Food` | `GF Food` |
| `GF Bev` | `GF Bev` |
| `Snack/Bev` | `Snack/Bev` |
| `Food` | `GF Food` — verify this |
**FILL_IN**: Any other class consolidations?
-
---
## 6. Remote Pricing Status — Meanings
**Status: ✅ READY** (data analyzed)
### Distribution
| Status | Count | Meaning |
| --- | --- | --- |
| **On** | 614 | Remote pricing is active and working normally |
| **Action Required** | 718 | Remote pricing needs attention — may have failed, needs configuration, or other issue |
| **Incompatible** | 516 | Machine does not support remote pricing (older model, EEEPROM Upgrade Required) |
**FILL_IN**: Clarify what each status actually means in practice:
**On** — Working
**Action Required** — No Dex Passcode / Bad Coil Count
**Incompatible** — achine does not support remote pricing (older model, EEEPROM Upgrade Required
Are there any other status values not present in this data?
---
---
## 7. Telemetry ID → Card Reader Decoder
**Status: ✅ READY** (data analyzed)
### Telemetry Provider Patterns
| Telemetry ID Pattern | Provider | Card Reader Type |
| --- | --- | --- |
| `VJ... (ePort)` | ePort (USI/Crane) | ePort telemetry module |
| `K3CT... (ePort)` | ePort (Crane) | ePort telemetry module |
| `VK... (ePort)` | ePort | ePort telemetry module |
| `... (NAYAX)` | Nayax | Nayax card reader |
| `... (CMS)` | Crane Merchandising Systems | CMS telemetry/reader |
| All-numeric (no suffix) | Unknown | Unknown/other |
### Distribution
| Provider | Count |
| --- | --- |
| ePort | 924 |
| CMS | 628 |
| Nayax | 275 |
| Other/unknown | 21 |
**FILL_IN**: More detail on what each provider means for card readers:
**ePort** —Looknuo cantaloupe devices_______________________________________________________________
**NAYAX** — look up nayax devices_______________________________________________________________
**CMS** — Crane integrated and external solutions_________________________________________________________________
**FILL_IN**: Do these map to specific credit card reader models?
| Provider | Reader Brand | Reader Model |
| --- | --- | --- |
| ePort | Cantaloupe___________ | ___________ |
| NAYAX | Nayax___________ | ___________ |
| CMS | Crane___________ | ___________ |
---
## 8. Alert Meanings
**Status: ✅ READY** (data analyzed)
### Alert Types Found
| Alert Text | Count | Severity | Meaning |
| --- | --- | --- | --- |
| `(empty)` | majority | none | No alerts |
| `This machine has been scheduled for service on Monday, May 25, 2026` | many | ⚠️ info | Scheduled maintenance |
| `This machine is scheduled for service today` | several | ⚠️ info | Same-day service scheduled |
| `Out of touch for 19 hours - contact Crane Merchandising Systems for assistance` | several | 🔴 critical | Telemetry offline > 19 hours |
| `Out of touch for 19 hours...` (combined with service notice) | few | 🔴 critical | Multiple issues |
**FILL_IN**: Categorize these alerts properly. What severity should each get?
| Alert | Severity (Info/Warning/Critical) | Display in app? |
| --- | --- | --- |
| Scheduled service in future | ___________ | ___________ |
| Scheduled service today | ___________ | ___________ |
| Out of touch (telemetry lost) | ___________ | ___________ |
| Out of touch + service needed | ___________ | ___________ |
**FILL_IN**: Are there other alert patterns that could appear?
---
---
## 9. Sales Priority Scoring
**Status: ⚠️ NEEDS INPUT**
The current `import_seed.csv` has **no sales data columns** — theyre marked as “Deleted” in the header map.
### Available Alternative Signals
Without sales data, we can score priority using:
| Signal | What it tells us | Priority hint |
| --- | --- | --- |
| `Class = "GF Food"` | Perishable food — needs frequent service | Higher priority |
| `Class = "GF Bev"` | Perishable drinks — needs frequent service | Higher priority |
| `Remote Pricing = "Action Required"` | Needs technical attention | Higher priority |
| `Alerts = "Out of touch"` | Telemetry down | Higher priority |
| `Customer = Disney` | High-traffic, high-visibility | Higher priority |
| `Type = "Unknown"` | Unknown machine — needs identification | Lower priority |
**FILL_IN**: How do you want to score priority?
| Criteria | Priority (Low / Mid / High) |
| --- | --- |
| GF Food or GF Bev class | ___________ |
| Action Required pricing | ___________ |
| Critical alert (out of touch) | ___________ |
| Disney property | ___________ |
| Active pricing (On) + Snack class | ___________ |
| Unknown Type/Make/Model | ___________ |
| Incompatible pricing | ___________ |
**FILL_IN**: Do you have sales data from a separate export? Where does it come from?
---
**FILL_IN**: If sales data is available, what are the priority thresholds?
| Annual Sales Range | Priority |
| --- | --- |
| $_______ - $_______ | Low |
| $_______ - $_______ | Mid |
| $_______+ | High |
---
## 10. Asset List & Detail Screen Spec
**Status: ⚠️ NEEDS FILL_IN**
### Asset List Card
Each asset card in the main app list should show:
**FILL_IN**: What fields belong on the asset card?
- [ ] Machine ID
- [ ] Name
- [ ] Make / Model
- [ ] Location (place, building, floor)
- [ ] Priority badge (low/mid/high)
- [ ] Status badge (active/maintenance/retired)
- [ ] Remote Pricing status
- [ ] Disney park badge
- [ ] Alert indicator
- [ ] GPS coordinates badge (has/dont have)
- [ ] Last contact / last dex time
- [ ] Card reader type
- [ ] Other: _______________
**FILL_IN**: Priority badge colors?
| Priority | Color |
| --- | --- |
| Low | ___________ |
| Mid | ___________ |
| High | ___________ |
### Asset Detail Screen
Full detail view should show:
**FILL_IN**: Section layout for the detail screen (order and contents):
1.
2.
3.
4.
5.
6.
**FILL_IN**: Which sections should be editable?
-
-
**FILL_IN**: Should the approval workflow fields be editable here too?
---
---
## 11. Update Policy — Per-Field
**Status: ⚠️ NEEDS FILL_IN**
For each field, specify how updates from new seed data should be handled:
| DB Column | Source CSV Column | Update Policy | Notes |
| --- | --- | --- | --- |
| `machine_id` | Machine ID | preserve | Primary key, never changes |
| `serial_number` | Serial Number | ___________ | |
| `name` | — (generated) | ___________ | |
| `make` | Type→Make (parsed) | ___________ | |
| `model` | Type→Model (parsed) | ___________ | |
| `class`*) | Type→Class (parsed) | ___________ | |
| `company` | Customer | ___________ | |
| `place` | Place (parsed) | ___________ | |
| `building_name` | Place (parsed) | ___________ | |
| `building_number` | Place (parsed) | ___________ | |
| `floor` | Place (parsed) | ___________ | |
| `room` | Place (parsed) | ___________ | |
| `trailer_number` | Place (parsed) | ___________ | |
| `address` | Address | ___________ | |
| `location_area` | City | ___________ | |
| `latitude` | — (derived) | preserve | Field-collected is authoritative |
| `longitude` | — (derived) | preserve | Field-collected is authoritative |
| `photo_path` | — | preserve | Never overwrite |
| `dex_report_date` | Last Dex Report Time | ___________ | |
| `install_date` | Added Date | ___________ | |
| `deployed` | — | ___________ | |
| `pulled_date` | — | ___________ | |
| `disney_park` | Customer (parsed) | ___________ | |
| `remote_pricing_status` | Remote Pricing | ___________ | |
| `alerts` | Alerts | ___________ | |
| `telemetry_provider` | Telemetry ID (parsed) | ___________ | |
| `card_reader_brand` | Telemetry ID (parsed) | ___________ | |
| `card_reader_model` | Telemetry ID (parsed) | ___________ | |
| `priority` | — (scored) | ___________ | |
| `prepick_group` | Prepick Group | ___________ | |
- `class` is not a current DB column — needs to be added via migration.
**Policy values:** `auto` (always overwrite), `approval` (queue for review), `preserve` (never touch)
**FILL_IN**: Add or change any policies above.
---
## Appendix: Column Mapping
### Raw Excel → CSV → DB Mapping
| Xlsx Column | CSV Column | DB Column | Notes |
| --- | --- | --- | --- |
| Device | Telemetry ID | — | Parsed to provider/brand |
| Location | Location | — | Redundant (Customer + Address) |
| Asset ID | Machine ID | `machine_id` | Primary key |
| Place | Place | `place` + derived fields | Main extraction target |
| Type | Type | — | Parsed to Class/Make/Model |
| City | City | `location_area` | Direct map |
| Address | Address | `address` | Direct map |
| Last Contact Time | Last Contact Time | — | Freshness indicator |
| Last Dex Report Time | Last Dex Report Time | `dex_report_date` | |
| Prepick Group | Prepick Group | — | Routing info |
| Customer | Customer | `company` | Direct map |
| Route | Route | — | Routing info |
| State | State | — | |
| Postal Code | Postal Code | — | |
| Serial Number | Serial Number | `serial_number` | Validate + correct |
| Class | Class | — | Source for `class` |
| Make | Make | `make` (fallback) | Type column takes priority |
| Model | Model | `model` (fallback) | Type column takes priority |
| Added Date | Added Date | `install_date` | |
| Status | Remote Pricing | — | Direct to app |
| Alerts | Alerts | — | Parsed to structured alerts |
**FILL_IN**: Any missing columns or corrections?
---