shawn/canteen-asset-tracker
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: MSFS data import pipeline — extraction → merge → assets.db

Browse Source
This commit is contained in:
Shawn
2026-05-28 22:07:14 -04:00
parent 77ffb0ab2a
commit b07c30da5e
1 changed files with 325 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/msfs-data-import.md
+325
View File
@@ -0,0 +1,325 @@
# MSFS Data Import — End-to-End Pipeline
> How Microsoft Dynamics 365 Field Service data was extracted, merged with Cantaloupe data, and imported into the Canteen Asset Tracker.
>
> **Date:** 2026-05-28
> **Assets in DB:** 9,636 (up from 1,848)
> **Assets with GPS:** 28
> **Backup saved:** `assets.db.20260528_214316.pre-msfs-import`
---
## 1. Extraction from Dynamics 365
The Dynamics 365 Field Service mobile app (Android) on a rooted Pixel 4a was the data source. The offline SQLite databases stored on the device contain full schema snapshots — `msdyn_customerasset`, `msdyn_workorder`, `account`, `bookableresourcebooking`, and 167+ other tables.
### 1.1 Phone → Disk
```bash
# Pull the main offline DB from the device
adb shell "su -c cp /data/data/com.microsoft.crm.crmphone.fieldServices/databases/msfs_backup.db /sdcard/"
adb pull /sdcard/msfs_backup.db data-samples/
# 1.7 GB SQLite database, pulled 2026-05-28
```
### 1.2 Dynamics 365 API Extraction (Live)
A Service Principal application was registered in Entra ID (same tenant as Field Service) with client credentials flow. The `dynamics_token.py` manager auto-refreshes tokens stored in `pass`.
```bash
# Get a token and pull equipment, accounts, work orders via Dataverse WebAPI
python3 dynamics_token.py # Outputs Bearer token
python3 query_booking_gps.py # Booking GPS from live API
```
**Results from live API:**
| Entity | Records |
|---|---|
| Equipment (`msdyn_customerasset`) | 14,440 |
| Accounts (`account`) | 5,305 |
| Work Orders (`msdyn_workorder`) | 15,531 |
### 1.3 Photo Extraction
3,703 technician photos were extracted from the offline DB's `annotation` / `activitymimeattachment` binary blobs to disk at `web/static/photos/`. These are actual field photos taken by technicians on their phones, containing:
- **EXIF GPS** from the phone camera (176 photos)
- **ConnectID stickers** and barcode images
- Work order photos of machines at customer sites
---
## 2. Merge (MSFS + Canteen)
### 2.1 The Merge Script
`standalone-merge` joins MSFS and Canteen records by **serial number** (`hsl_serialnumber`). No other common key exists.
**Source tables from MSFS:**
- `msdyn_customerasset` — equipment records with serials, GPS, manufacturer, model, ConnectID
- `msdyn_workorder` — work order history linked to assets
- `account` — customer account details with geocoded addresses
**Source from Canteen:**
- Old `assets.db` (1,848 records from Cantaloupe CSV import)
### 2.2 Join Logic
```python
# standalone-merge → merge()
all_serials = set(msfs_assets) | set(canteen_assets)
for serial in sorted(all_serials):
if msfs and canteen: match = "joined" # Both sources have this asset
elif msfs: match = "msfs_only" # Only MSFS knows about it
else: match = "canteen_only" # Only Cantaloupe import has it
```
Fields are prefixed by source: `msfs_*` for MS Field Service fields, `canteen_*` for Cantaloupe import fields. Work orders are attached per-asset.
### 2.3 Merge Results
| Metric | Count |
|---|---|
| **Total merged records** | 10,038 |
| **Joined** (both sources, matched by serial) | 1,832 |
| **MSFS-only** (no Cantaloupe match) | 8,204 |
| **Canteen-only** (no MSFS match) | 2 |
| **Work orders linked** to assets | 9,652 |
| **Assets with work orders** | 3,199 |
| **Accounts resolved** | 3,486 |
| **Customers resolved** | 284 |
| **Locations resolved** | 364 |
**Timing:** The "MSFS-only" dominance means the old Cantaloupe import had very limited coverage. Most of the 8,204 MSFS-only assets are from Dynamics-only equipment not managed in the Cantaloupe system.
---
## 3. Field Mapping
### 3.1 GPS Resolution (Priority Order)
When importing into the assets table, GPS coordinates are resolved in this order:
1. **MSFS latitude/longitude** from `msdyn_customerasset` (if present)
2. **Canteen latitude/longitude** from old imports (if MSFS missing)
3. If both missing → `NULL` (no GPS — 9,608 assets fall here)
The GPS in `msdyn_customerasset` itself is **geocoded account addresses**, not real device GPS. Real EXIF GPS from technician photos would need the OCR pipeline (see §4).
### 3.2 Field Mapping Spec
| `assets` column | Source | MSFS field |
|---|---|---|
| `machine_id` | msfs_machine_id → canteen_machine_id | Extracted from `hsl_equipmentid` suffix |
| `serial_number` | serial_number | `hsl_serialnumber` |
| `name` | msfs_name → canteen_name | `msdyn_name` |
| `make` | msfs_manufacturer | `hsl_manufacturertext` |
| `model` | msfs_dex_model | `hsl_dexmodel` |
| `latitude` / `longitude` | msfs → canteen | `msdyn_latitude` / `msdyn_longitude` |
| `address` | account.address | From `account` table |
| `connect_id` | msfs_connect_id | `hsl_connectid` |
| `canteen_connect_guid` | msfs_canteen_connect_guid | `hsl_canteenconnectguid` |
| `manufacturer` | msfs_manufacturer | `hsl_manufacturertext` |
| `equipment_id` | msfs_equipment_id | `hsl_equipmentid` |
| `company` | canteen_company | Old Cantaloupe import |
| `category` | canteen_category | Old Cantaloupe import |
| `install_date` | canteen → msfs | `hsl_opendate` |
### 3.3 Output JSON
`web/static/data/merged-assets.json` — 10,038 records (159 MB) with full field mapping, work order attachments, account/customer/location resolution, and GPS coordinates.
---
## 4. OCR & Photo EXIF GPS Pipeline
### 4.1 Problem
- 9,608 of 9,636 assets have no real GPS
- Existing GPS is geocoded addresses (account-based, not actual machine location)
- 176 photos contain real EXIF GPS from technician phone cameras
### 4.2 Approach
The plan in `docs/plans/2026-05-28-photo-exif-gps-pipeline.md` outlines a multi-pass OCR pipeline:
1. **Tesseract OCR** on all 3,703 photos — 4-tier cascade (scales, preprocessing modes, PSM variants)
2. **Vision API fallback** (`mimo-v2-omni`) for Tesseract failures
3. **Machine ID extraction** from ConnectID stickers (`XXXXX-YYYYYY`), serial plates, barcodes
4. **Cross-reference** OCR results with photo EXIF GPS
5. **Write high-confidence** updates to `assets.db` with `gps_source = 'photo_exif'`
**Current status:** 201 photos OCR'd as proof of concept (5.4%), 14 unique machine IDs identified.
### 4.3 Key Scripts
| Script | Purpose |
|---|---|
| `ocr_local.py` | Tesseract 4-tier local OCR cascade |
| `ocr_batch.py` | Vision API batch OCR (OpenCode Go) |
| `ocr_mapping.py` | OCR results → machine ID → canteen asset mapping |
| `consolidate_gps.py` | Multi-source GPS consolidation |
| `_ocr_analyze.py` | OCR failure analysis |
| `standalone-merge` | MSFS → Canteen merge |
---
## 5. Import Script
### `import_msfs.py`
The stand-alone import script located at `~/projects/canteen-asset-tracker/import_msfs.py` performs:
1. **Backup** existing `assets.db` → `assets.db.{timestamp}.pre-msfs-import`
2. **Read** `merged-assets.json` (10,038 records)
3. **Create fresh** `assets.db` with full v2 schema (18 tables, indexes, triggers)
4. **Map** merged fields to assets table columns per the mapping spec above
5. **Deduplicate** by `machine_id` with priority: joined > msfs_only > canteen_only
6. **Insert** categories, customers, locations from merged data
7. **Insert** default users (admin/technician)
8. **Verify** with table/row/index counts
### Run
```bash
cd ~/projects/canteen-asset-tracker
python3 import_msfs.py
```
### DB Stats After Import
```sql
Assets total: 9,636
Assets with GPS: 28
Categories: 21
Customers: 284
Locations: 364
Users: 2 (admin, tech)
Table count: 18
Index count: 6
DB size: ~11 MB (WAL mode)
```
---
## 6. Extraction DB (Live Route Planning)
The extraction DB is linked from the canteen asset tracker server for live work order and route optimization queries:
```python
# server.py → _get_extraction_db()
EXTRACTION_DB = (
Path.home()
/ "projects/ms-field-service-extraction"
/ "data-samples/msfs_backup"
/ "fieldservicecanteen.crm.dynamics.com_9fc6c50c-b097-f011-b4cb-7ced8d1b15a2_data.db"
)
```
Five endpoints use this:
| Endpoint | Purpose |
|---|---|
| `GET /api/workorders/search` | Paginated search by name/account/city |
| `POST /api/workorders/lookup` | Bulk lookup by work order ID/name |
| `GET /api/workorders/today` | Today's active bookings by technician |
| `GET /api/workorders/technicians` | Distinct technician list |
| `POST /api/route/optimize` | TSP route optimization (nearest-neighbor + 2-opt) |
---
## 7. File Locations
| Artifact | Path |
|---|---|
| MSFS backup DB | `~/projects/ms-field-service-extraction/data-samples/msfs_backup/` |
| Merged JSON | `~/projects/ms-field-service-extraction/web/static/data/merged-assets.json` |
| Live API data | `data-samples/dynamics_equipment.json` (14,440), `dynamics_accounts.json` (5,305), `dynamics_workorders.json` (15,531) |
| Photos | `~/projects/ms-field-service-extraction/web/static/photos/` (3,703 files) |
| OCR results | `web/static/data/ocr_results.json` |
| Merge script | `~/projects/ms-field-service-extraction/standalone-merge` |
| Import script | `~/projects/canteen-asset-tracker/import_msfs.py` |
| Canteen SQLite DB | `~/projects/canteen-asset-tracker/assets.db` |
| Pre-import backup | `assets.db.20260528_214316.pre-msfs-import` |
| GPS consolidation | `~/projects/ms-field-service-extraction/consolidate_gps.py` |
| GPS consolidated JSON | `web/static/data/gps_consolidated_update.json` |
| Data catalog | `~/projects/ms-field-service-extraction/DATA_CATALOG.md` |
| Photo EXIF GPS plan | `docs/plans/2026-05-28-photo-exif-gps-pipeline.md` |
---
## 8. Architecture Diagram
```
┌─────────────────────────────────────┐
│ Rooted Pixel 4a │
│ (Shawn Canada's device) │
│ MS Field Service Mobile App │
│ Offline SQLite (1.7 GB, 167+ tbls) │
└──────────────┬──────────────────────┘
│ adb pull
▼
┌─────────────────────────────────────┐
│ MSFS Backup DB │
│ data-samples/msfs_backup/ │
│ - msdyn_customerasset (14K rows) │
│ - msdyn_workorder (15K rows) │
│ - account (5K rows) │
│ - annotation (photos, 3.7K blobs) │
└──────────┬──────────────────────────┘
│
┌────────────────┼────────────────────┐
▼ ▼ ▼
┌─────────────────┐ ┌─────────────┐ ┌─────────────────┐
│ photo extraction │ │ merge │ │ Dynamics API │
│ 3,703 JPEGs │ │ standalone │ │ (Service Prin.) │
│ EXIF GPS → 176 │ │ -merge │ │ 14K equipment │
│ → OCR pipeline │ │ serial join │ │ 15K work orders │
└────────┬─────────┘ └──────┬──────┘ └─────────────────┘
│ │
▼ ▼
┌─────────────────────────────────────────────┐
│ merged-assets.json │
│ 10,038 records │
│ 1,832 joined / 8,204 MSFS-only │
│ 9,652 work orders linked │
└──────────────────┬──────────────────────────┘
│ import_msfs.py
▼
┌─────────────────────────────────────────────┐
│ assets.db │
│ 9,636 assets (was 1,848) │
│ 28 with GPS │
│ 18 tables │
│ 6 indexes │
└──────────────────┬──────────────────────────┘
│ server.py links for
│ live route/work order
▼
┌─────────────────────────────────────────────┐
│ Canteen Asset Tracker App │
│ → Find Asset (barcode/OCR lookup) │
│ → Map (Leaflet, GPS pins) │
│ → Route (TSP optimization, extraction DB) │
│ → Nav (OSRM driving/walking) │
└─────────────────────────────────────────────┘
```
---
## 9. Future Work
The **Photo EXIF GPS Pipeline** (see `docs/plans/2026-05-28-photo-exif-gps-pipeline.md`) is designed to add real GPS to the remaining 9,608 assets:
| Task | Description | Status |
|---|---|---|
| 0 | Diagnose OCR bottleneck (why only 14/3,703 photos yield IDs) | ⬜ |
| 1 | Fix OCR regex for ConnectID, serial, barcode formats | ⬜ |
| 2 | Batch OCR all 3,703 photos with Tesseract (all photos, not just GPS) | ⬜ |
| 3 | Vision API OCR on Tesseract failures | ⬜ |
| 4 | Multi-source disambiguation (timestamps + location) | ⬜ |
| 5 | Push verified GPS into `assets.db` with source tracking | ⬜ |
| 6 | Verify and visualize on map | ⬜ |
Each photo with EXIF GPS that can be matched to a machine ID gives us a **real device GPS coordinate** — far more accurate than the current geocoded addresses.