From 27cfa6d68e370adb5a1b81ee1af80947a2d2eafe Mon Sep 17 00:00:00 2001 From: Shawn Date: Tue, 2 Jun 2026 22:51:05 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20add=20FIELD=5FMAP.md=20=E2=80=94=20comp?= =?UTF-8?q?lete=20field=20origin=20reference=20for=20all=20pipelines?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/FIELD_MAP.md | 225 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 225 insertions(+) create mode 100644 docs/FIELD_MAP.md diff --git a/docs/FIELD_MAP.md b/docs/FIELD_MAP.md new file mode 100644 index 0000000..5731ffe --- /dev/null +++ b/docs/FIELD_MAP.md @@ -0,0 +1,225 @@ +# Field Map — Canteen Asset Tracker + +Every column in `assets.db`, how each pipeline touches it. + +## Source Legend + +| Source | Pipeline | File(s) | How to Run | +|--------|----------|---------|------------| +| **MSFS** | ADB pull → Dynamics 365 extraction | `pull_msfs_data.py` → `sync_assets_to_canteen.py` | Button on extraction viewer `:8910` **Manual** | +| **CANT** | Cantaloupe staged import | `cantaloupe_sync.py` in admin server | Dashboard upload **Manual** | +| **SEED** | Seed Excel import | `parser.py` → `db_writer.py` → `sync.py` | CLI `--mode=seed` or `--mode=update` **Manual** | +| **APP** | Main app (technician check-in, photo, GPS) | `server.py` | Used by techs in field | +| **ADMIN** | Admin dashboard edits | `admin_server.py` | Dashboard Cleanup/Mass Edit | + +## Policy Legend + +- **always** — pipeline overwrites this field every run +- **if_empty** — only filled when currently NULL/empty +- **never** — pipeline never touches this column +- **if_numeric** — only overwritten if value is bare number (Cantaloupe default ID) + +--- + +## 🔵 MSFS Sync — Field Policy + +| Column | Policy | Dynamics 365 Source | +|--------|--------|---------------------| +| `serial_number` | always | `hsl_serialnumber` | +| `connect_id` | always | `hsl_connectid` | +| `equipment_id` | always | `hsl_equipmentid` | +| `canteen_connect_guid` | always | `hsl_canteenconnectguid` | +| `barcode` | always | `hsl_barcode` | +| `manufacturer` | always | `hsl_manufacturertext` | +| `model` | always | `hsl_modeltext` or `hsl_modelname` | +| `route_name` | always | `hsl_routeidname` | +| `install_date` | always | `hsl_opendate` | +| `device` | always | `msdyn_deviceid` | +| `name` | if_numeric | `msdyn_name` or `hsl_name` | +| `company` | if_empty | Account `name` | +| `customer_name` | if_empty | Account `name` | +| `place` | if_empty | Account `address1_city` | +| `address` | if_empty | Account `address1_line1` + `line2` | +| `make` | **never** | _(uses `manufacturer` → separate column)_ | +| `building_name` | **never** | — | +| `floor` | **never** | — | +| `room` | **never** | — | +| `building_number` | **never** | — | +| `trailer_number` | **never** | — | +| `location_area` | **never** | — | +| `latitude` | **never** | _(mapped but only if NULL in code)_ | +| `longitude` | **never** | _(mapped but only if NULL in code)_ | +| `photo_path` | **never** | — | +| `category` | **never** | — | +| `status` | **never** | — | +| `description` | **never** | — | + +> All other columns (sales data, alerts, disney flags, etc.) are **never** touched by MSFS sync. + +--- + +## 🟡 Cantaloupe Sync — Field Policy + +| Column | Policy | Notes | +|--------|--------|-------| +| `machine_id` | never (match key) | — | +| `serial_number` | always | — | +| `name` | always | — | +| `description` | always | — | +| `category` | always | Validated against categories table | +| `status` | always | Validated against statuses | +| `make` | always | — | +| `model` | always | — | +| `address` | always | — | +| `building_name` | always | — | +| `building_number` | always | — | +| `floor` | always | — | +| `room` | always | — | +| `trailer_number` | always | — | +| `walking_directions` | always | — | +| `map_link` | always | — | +| `parking_location` | always | — | +| `photo_path` | always | — | +| `latitude` | always | — | +| `longitude` | always | — | +| `geofence_radius_meters` | always | — | +| `dex_report_date` | always | — | +| `deployed` | always | — | +| `pulled_date` | always | — | +| `customer_id` | always | Via customer name lookup/creation | +| `location_id` | always | Via location name lookup/creation | +| `is_disney` | always | Derived from customer name prefix `D-` | + +> **No preserve logic.** Any non-empty field in the import file overwrites the existing value. + +--- + +## 🟢 Seed Import — Field Policy (db_writer.py) + +### Seed Mode (`--mode=seed`) +**DESTROYS AND RE-INSERTS** — `DELETE FROM assets` first, then inserts all rows fresh. + +### Update Mode (`--mode=update`) +Upsert by `machine_id`. **MSFS is primary data source** — seed fills gaps only. + +**ALL fields** are treated as ONLY_IF_EMPTY for existing records: +| Column | Policy | Excel Source / Generation | +|--------|--------|--------------------------| +| `serial_number` | if_empty | Serial Number column + OCR corrections | +| `name` | if_empty | Generated: `{make} @ {place}` | +| `make` | if_empty | Parsed from Type column | +| `model` | if_empty | Parsed from Type column | +| `class` | if_empty | Parsed from Type column | +| `is_glass_front` | if_empty | Parsed from Type column | +| `company` | if_empty | Parsed from Customer column | +| `place` | if_empty | Parsed from Place column | +| `building_name` | if_empty | Parsed from Place column | +| `suite` | if_empty | Parsed from Place column | +| `room` | if_empty | Parsed from Place column | +| `trailer` | if_empty | Parsed from Place column | +| `floor` | if_empty | Parsed from Place column | +| `zone` | if_empty | Parsed from Place column | +| `zone_type` | if_empty | Parsed from Place column | +| `address` | if_empty | Address column | +| `location_area` | if_empty | City column | +| `latitude` | if_empty | Excel (existing GPS preserved) | +| `longitude` | if_empty | Excel (existing GPS preserved) | +| `photo_path` | if_empty | Excel (existing photo preserved) | +| `install_date` | if_empty | Added Date (existing date preserved) | +| `deployed` | if_empty | Deployed column (existing preserved) | +| `pulled_date` | if_empty | Pulled Date (existing preserved) | +| `dex_report_date` | if_empty | Last Dex Report Time | +| `disney_park` | if_empty | Parsed from Customer column | +| `disney_area` | if_empty | Derived from disney_park | +| `remote_pricing_status` | if_empty | Status column | +| `alerts` | if_empty | Alerts column + Coil/Product Alerts | +| `telemetry_provider` | if_empty | Parsed from Device column | +| `card_reader_brand` | if_empty | Parsed from Device column | +| `priority` | if_empty | Computed from yearly_sales | +| `yearly_sales` | if_empty | Yearly Sales column | +| `monthly_sales` | if_empty | Monthly Sales column | +| `weekly_sales` | if_empty | Weekly Sales column | +| `days_since_restock` | if_empty | Days Since Restock column | +| `prepick_group` | if_empty | Prepick Group column | +| `has_cashless` | if_empty | Has Cashless column | + +> **New assets** (not found in DB by machine_id) are inserted with full seed data. +> **Existing assets**: only empty/null fields are filled. Any field already populated by MSFS, cleanup, or field use is preserved. + +--- + +## 🟠 Seed Import — Sync (`sync.py` → main app DB) + +After `db_writer.py`, this script pushes the seed DB into the main app's schema. +**Now follows ONLY_IF_EMPTY policy** for existing assets — same as db_writer update mode. + +| Column | Policy | +|--------|--------| +| `machine_id` | never (match key) | +| `serial_number` | if_empty | +| `name` | if_empty | +| `description` | if_empty | +| `category` | if_empty | +| `status` | if_empty | +| `make` | if_empty | +| `model` | if_empty | +| `address` | if_empty | +| `building_name` | if_empty | +| `building_number` | if_empty | +| `floor` | if_empty | +| `room` | if_empty | +| `trailer_number` | if_empty | +| `walking_directions` | if_empty | +| `map_link` | if_empty | +| `parking_location` | if_empty | +| `photo_path` | if_empty | +| `latitude` | if_empty | +| `longitude` | if_empty | +| `geofence_radius_meters` | if_empty | +| `disney_park` | if_empty | +| `is_disney` | if_empty | +| `dex_report_date` | if_empty | +| `install_date` | if_empty | +| `deployed` | if_empty | +| `pulled_date` | if_empty | +| `customer_id` | if_empty (via upsert customers table) | +| `location_id` | if_empty (via upsert locations table) | + +> **Also creates/updates** `customers` and `locations` tables. + +--- + +## ⚪ Admin Tools — Policy + +### Batch Update (Cleanup page) +- `make`, `model`, `name`, `company`, `place`, `building_name`, `building_number`, `floor`, `room`, `location_area`, `address`, `category`, `status`, `customer_name` +- **Always overwrites** with user-provided value. + +### Clear GPS +- `latitude`, `longitude` — set to NULL + +### Mass Edit (existing tool) +- Same allowed fields as Batch Update. + +--- + +## ⚫ Main App (Tech Field) — What Touches What + +| Action | Fields Affected | +|--------|----------------| +| Barcode scan check-in | `updated_at` | +| GPS check-in | `latitude`, `longitude`, `updated_at` | +| Photo upload | `photo_path`, `updated_at` | +| OCR scan (name search) | (read-only) | +| Register new asset | All columns with defaults | + +--- + +## Pipeline Activity Status + +| Pipeline | Currently Scheduled? | Frequency | Risk to Cleanup | +|----------|--------------------|-----------|-----------------| +| **MSFS Sync** | 🔴 **Manual only** | On-demand via extraction viewer | Low — preserves name/company/place/building/floor. Only `model` overwritten. | +| **Cantaloupe Sync** | 🔴 **No cron** | Manual upload on admin dashboard | High — no preserve logic. Don't run after cleanup (or we fix it too). | +| **Seed Import** `--mode=update` | 🔴 **Manual CLI only** | `python3 main.py -m update` | ✅ **Safe now** — ONLY_IF_EMPTY. Fills gaps, doesn't overwrite MSFS/cleanup data. | +| **Seed Import** `--mode=seed` | 🔴 **Manual CLI only** | `python3 main.py -m seed` | 🔥 **NUCLEAR** — `DELETE FROM assets`. Full re-insert. |