Stage 0 -- Research Brief

# Stage 0 -- Research Brief
## K11 Production System Architecture

_Ground truth gathered 2026-04-27 from codebase, docs, and external references._

---

1. What Exists Today

### Hardware (on hand or arriving today)
- GMKtec NucBox K11: Ryzen 9 8945HS, Radeon 780M (~9 TFLOPS), 32GB DDR5, 1TB NVMe, Windows 11 Pro. 4x USB-A, HDMI 2.1, DP 2.1, 2x USB4-C, dual 2.5G NIC, WiFi 6, BT 5.x. Physical: 131.83 x 124.46 x 57.91mm (caliper-verify pending).
- Orbbec Femto Bolt: USB-C depth camera, ~140x39x30mm. Arriving Apr 27.
- Orbbec Femto Mega: USB-C depth camera with onboard Jetson, ~127x65x38mm. Already on hand.
- Sony mocopi: 6 BLE IMU sensors (hip, wrists, ankles, head). Already on hand.
- UMA-8 microphone array: USB audio interface. Already on hand.
- 3D-printed T-form pod: 200x130x165mm rear pod behind 500x120x85mm bar. K11 + camera + cables inside.

### Software -- Python Publishers (software/demo/)
| File | Lines | Role | Port | Wire Format |
|------|-------|------|------|-------------|
| `pointcloud_pub.py` | 556 | Depth/cloud publisher | :9700 | LUME (0x4C554D45) + LUMD (0x4C554D44) |
| `audio_pub.py` | 315 | Audio FFT sidecar | :9701 | LUMF (0x4C554D46, 84 bytes fixed) |
| `lume_packet_inspector.py` | 229 | Debug/diagnostics | listens :9700+:9701 | reads all |
| `mocopi_bridge.py` | DOES NOT EXIST | Referenced by start-mocopi.bat | :9702 | LUMM (undefined) |

### Software -- Unity Components (Assets/Scripts/, 2561 total lines)
| Component | Lines | Exec Order | Role |
|-----------|-------|-----------|------|
| LumeUdpReceiver | 234 | bg thread | UDP socket, magic-byte dispatch |
| LumePointRenderer | 120 | default | URP instanced rendering |
| LumeDepthReprojector | 198 | default | GPU pinhole projection |
| LumeOpticalFlow | 312 | default | Dense LK + frame-diff scalar |
| LumeAudioReactor | 138 | 0 | Mic/idle-sine fallback |
| LumeMotionGate | 110 | 150 | SuperHot time-scaling |
| LumeAudioFftReceiver | 283 | 200 | LUMF decode + shader override |
| LumeVfxRuntimeBridge | 144 | 210 | VFX Graph param push |
| LumeTransientForcePusher | 153 | 220 | Beat-edge VFX events |
| LumeCalibrationPanel | 271 | 0 | F12 runtime IMGUI tweaks |
| LumeProceduralTunnel | 179 | default | 6144-pt fallback helix |

### Software -- MotionMix iOS App
| File | Lines | Role |
|------|-------|------|
| MocopiReceiver.swift | 240 | UDP/OSC listener on :9500, parses 27-bone skeleton |
| MocopiFeatureExtractor.swift | 233 | Compresses 27 bones to 24D features for SAN 128D vector |

### Software -- Mac Service Management
- `com.lume.synthpub-mac5.plist`: LaunchAgent, KeepAlive, /usr/bin/python3
- `com.lume.audio-pub-mac5.plist`: LaunchAgent, KeepAlive, /usr/bin/python3
- `install-mac5.sh`: Copies scripts to [home-path] bootstraps agents
- `health-check.sh`: One-shot diagnostic (LaunchAgent status, port probes, log tails)

### Software -- K11 Windows Scripts (scripts/k11/)
- `k11-bootstrap.ps1`: Admin PowerShell (enables RDP, installs Git/Python/Unity Hub/OBS)
- `start-lume-pipeline.bat`: Launches audio_pub + pointcloud_pub + inspector as named windows
- `start-femto-depth.bat`: Kills synthetic, starts real Femto depth publisher
- `start-mocopi.bat`: References nonexistent `mocopi_bridge.py` on :9702

### Wire Formats (lume_wire_format.md)
| Magic | Name | Header | Port | Status |
|-------|------|--------|------|--------|
| 0x4C554D45 | LUME (cloud) | 16B | 9700 | Shipped |
| 0x4C554D44 | LUMD (raw depth) | 40B | 9700 | Shipped |
| 0x4C554D46 | LUMF (audio) | 36B + 48B body = 84B | 9701 | Shipped |
| 0x4C554D43 | LUMC (color) | reserved | 9700 | Not implemented |
| 0x4C554D4D | LUMM (mocopi) | undefined | 9702 | Does not exist |

### Existing mocopi Integration (MotionMix iOS)
- MocopiReceiver.swift listens on UDP :9500 for OSC-formatted bone data
- Parses 3 OSC formats: /mcp/sklt/joint/N [7 floats], /mocopi/skel [189 floats flat], /mcp/BonePos/{name} [3 floats]
- 27 bones, each [x,y,z, qw,qx,qy,qz] or position-only
- Routes to MocopiFeatureExtractor.update(bones:) for 24D compression
- Sony mocopi app sends from phone to PC via UDP on port 12351 (configurable)
- Sony's protocol uses custom binary format (NOT OSC) with skdf (skeleton definition) + fram (frame data) packet types

### Sony mocopi Receiver Plugin for Unity
- Official plugin: github.com/sony/mocopi-receiver-plugin-unity
- Requires Unity 6000.0.60f1 or later (compatible with our 6000.3.2f1)
- Receives mocopi data directly via UDP on :12351
- Provides MocopiSimpleReceiver prefab + Mocopi Avatar component
- Supported on Windows 10/11, macOS 10.15+
- Does NOT support Linux

---

2. What's Been Tried Before

### Mac-based LUME pipeline (Waves 1-5, shipped)
- LaunchAgent-based auto-start on Mac5 works reliably
- KeepAlive + ThrottleInterval handles crashes gracefully
- install-mac5.sh pattern: copy script to [home-path] install plist, bootstrap
- health-check.sh pattern: probe LaunchAgents, lsof ports, tail logs, 5s inspector snapshot

### K11 bootstrap scripts (just created, untested)
- start-lume-pipeline.bat uses `start "TITLE" /min cmd /c` pattern
- References `mocopi_bridge.py` that doesn't exist
- No auto-restart on crash (bare `start` command)
- No health-check equivalent for Windows
- No service management (no NSSM, no Task Scheduler)

### MotionMix mocopi integration (shipped, iOS)
- OSC parsing works but only for mobile-app-to-iOS path
- The Sony mocopi official protocol (binary, not OSC) is different from what MocopiReceiver.swift parses
- The official Sony receiver Unity plugin handles the binary protocol natively
- 24D feature extraction from 27 bones is proven (MocopiFeatureExtractor.swift)

---

3. Real Constraints

### Hardware
- K11 has Radeon 780M (AMD iGPU). Unity compute shaders must work on Vulkan/DX12 (not Metal).
- 32GB DDR5 is shared between CPU and GPU (UMA). GPU allocation affects available system RAM.
- K11 fits in pod with ~1mm Z clearance (POD_H should be 130mm not 120mm).
- USB-C passthrough for Femto camera must be reliable under vibration.
- BT range for mocopi sensors: ~10m typical. Pod placement matters.
- Only 4 USB-A ports available (keyboard, mouse use 2 during setup; UMA-8 uses 1).

### Software
- Unity project currently uses macOS Metal shaders. GraphicsFormat.R16G16_SFloat (used in LumeOpticalFlow.cs) must be verified on Vulkan/DX12.
- Python publishers use macOS-specific UDP constants (net.inet.udp.maxdgram=9216). Windows has different defaults (65507 max datagram).
- LaunchAgent concepts (KeepAlive, ThrottleInterval, bootstrap/bootout) have NO direct Windows equivalent.
- Sony mocopi Windows PC app can receive BLE data directly (no phone needed). But mocopi sensors still pair via BT to a phone first for calibration.
- pyorbbecsdk (Orbbec Python SDK) Windows compatibility unverified.

### Network
- All UDP publishers bind to [ip] by default. For Tailscale mesh access, must bind to [ip].
- Windows Firewall blocks inbound UDP by default. Must add rules for :9700, :9701, :9702, :12351.
- Tailscale must be installed and authorized (covered in K11-FIRST-BOOT.md).
- MotionMix iOS app needs to discover K11's IP to receive mocopi data bridge.

### Service Management Options on Windows
1. NSSM (Non-Sucking Service Manager): Single .exe, wraps any console app as Windows service. Auto-restart, SCM integration, log rotation, runs before login. Most Mac LaunchAgent-like.
2. Task Scheduler: Built-in. Can run at startup, on login, on schedule. No auto-restart on crash without wrapper. More setup friction.
3. Startup Folder: Shell:startup shortcut. Runs after login only. No auto-restart. Simplest but weakest.
4. Windows Service (native C#/Python): Requires service-aware code. Most robust but highest dev cost.

---

4. Open Questions for Stage 1

1. Should Unity receive mocopi directly (Sony plugin) or via LUMM bridge? Sony's plugin exists and works on Unity 6000.x. But it's a separate data path outside the LUME wire format family. A bridge (mocopi_bridge.py) normalizes everything into the LUMM UDP format, keeping the architecture consistent.

2. Should mocopi run from phone or K11 directly? The Sony mocopi Windows PC app can pair BLE sensors directly. Eliminates the phone as middleman. But initial calibration still requires the phone app.

3. Which service manager? NSSM matches LaunchAgent behavior most closely (auto-restart, pre-login, SCM). Task Scheduler is simpler but weaker. Startup folder is fragile.

4. Should MotionMix iOS connect to K11 or should K11 push to MotionMix? Current architecture: mocopi phone app pushes to iOS MocopiReceiver. New architecture: K11 receives from mocopi directly, must somehow get data to MotionMix iOS.

5. How does the Unity project build for Windows? Current development is on macOS. Need to verify: compute shader compatibility (Metal -> Vulkan/DX12), Unity project settings, player build export.

6. How many Python processes run simultaneously? audio_pub + pointcloud_pub + mocopi_bridge + inspector = 4 Python processes. On Ryzen 9 8945HS with 32GB, this is trivial. But pyaudio + pyorbbecsdk + numpy all need Windows-compatible installs.

7. Pod thermal management? K11 at 65W TDP inside sealed pod with limited airflow. Need ventilation strategy or accept throttling.