Stage 3: EXPAND + MASTER PLAN

Stage 3: EXPAND + MASTER PLAN

3a. Risk Audit

CRITICAL RISKS

R1: Radeon 780M Compute Shader Compatibility
- Failure scenario: Unity compute shaders (LumeDepthReproject.compute, LumeOpticalFlow.compute) fail on AMD Vulkan/DX12 drivers. These were developed on Apple Metal (Mac5).
- Probability: 20
- Impact: HIGH (no visual pipeline without GPU reprojection)
- Mitigation: Test on K11 Day 2. If Vulkan fails, try DX12 backend. If both fail, use LUME format (CPU-reprojected in pointcloud_pub.py) instead of LUMD (GPU-reprojected). Wave 1 graceful-degrade path exists.
- Validation: `pointcloud_pub.py --synthetic-depth` + Unity running on K11 produces visible animated surface.

R2: Tailscale Mesh Reliability for LUMM/LUMF to iOS
- Failure scenario: Tailscale has documented 100
- Probability: 30
- Impact: HIGH (music disconnects from body motion)
- Mitigation: (1) iOS local fallback: if no LUMM received for 3s, fall back to local CoreMotion/Vision-only 128D (dims [76:100] stay zero, as they are today pre-LUMM). (2) LUMF fallback: if no LUMF for 3s, ambient audio dims [104:108] stay zero. (3) Health check: lume_packet_inspector.py on K11 logs drop rates. echelon_relay.py pings Tailscale targets on startup.
- Validation: Unplug Tailscale adapter on K11 while iOS is consuming. Verify music continues with degraded but not broken behavior.

R3: mocopi BLE Pairing Unreliability
- Failure scenario: Sony mocopi 6x sensors fail to pair via BLE to K11 Windows. BLE pairing on Windows is notoriously flaky. Without mocopi, LUMM :9702 is empty.
- Probability: 35
- Impact: MEDIUM-HIGH (lose skeleton input, but depth + audio still work)
- Mitigation: (1) Primary: use Sony mocopi app on phone, phone connects to sensors via BLE, phone sends to K11 via OSC :9500 instead of K11 BLE direct. mocopi_bridge.py already has an OSC listener path. (2) Secondary: skip mocopi entirely. Depth camera optical flow + audio transients still drive visuals. iOS still has CoreMotion/Vision for music.
- Validation: Run system for 30 minutes with only Femto Bolt + UMA-8. Verify visuals and music are engaging without skeleton data.

MEDIUM RISKS

R4: Music-to-Visual Synchronization Drift
- Failure scenario: K11 visuals react to local depth/audio sensors (~13ms latency). Mac5 audio reacts to iOS-processed data (~50ms). A hand clap is visible in depth at T=0 but audible at T=37ms later. Perceptible desync during fast transients.
- Probability: 50
- Impact: MEDIUM (subconscious unease, not show-stopping)
- Mitigation: (1) Local audio reactivity on K11 via LUMF means visuals DO react to the clap at T=8ms (via LumeAudioFftReceiver transient). Music follows at T=50ms. The visual leads the audio, which is more natural than the reverse. (2) For beat-synced visual events (section transitions, preset changes), use bar-aligned timing (1-4 second granularity) where 50ms jitter is imperceptible.
- Validation: Record video of LUME display + audio output simultaneously. Measure transient alignment. Accept if <100ms (perceptible sync window).

R5: Unity Main Thread Overload with New Components
- Failure scenario: Adding LumeMocopiReceiver, LumeSystemBus, LumeEchelonReceiver increases Unity main thread work. Drop below 60fps.
- Probability: 25
- Impact: MEDIUM (visual stutter)
- Mitigation: (1) New components are lightweight: LUMM is 772B/frame at 50Hz (parse only), LUMS is 64B/frame at 30Hz (write only), LUME_ECH is 48B/frame at 10Hz. Total added work: <0.1ms per frame. (2) Profile with Unity Profiler on K11 before declaring done.
- Validation: Unity Profiler shows frame time <12ms (leaving 4ms headroom for 60fps).

R6: echelon_relay.py WebSocket Stability
- Failure scenario: WebSocket connection to :9404/director/ws drops during performance. echelon_relay.py stops publishing LUME_ECH. Unity loses section/tension/tempo awareness.
- Probability: 30
- Impact: MEDIUM (visuals lose global coordination, revert to local-only reactivity)
- Mitigation: (1) Auto-reconnect with exponential backoff (1s, 2s, 4s, max 10s). (2) Unity LumeEchelonReceiver has stale-data timeout: if no LUME_ECH for 5s, revert to default values (_LumeSection=0, _LumeTension=0.5). (3) echelon_relay.py logs connection state to stderr.
- Validation: Kill multicam-server during relay operation. Verify relay reconnects within 10s. Verify Unity degrades gracefully.

R7: Sony Binary Protocol Parsing
- Failure scenario: Sony mocopi PC app uses different TLV structure than phone app. The existing mocopi-bridge.py was written for phone output. PC app may use different magic bytes, different bone ordering, or different coordinate system.
- Probability: 30
- Impact: MEDIUM (LUMM skeleton data is wrong or empty)
- Mitigation: (1) Hex dump first 10 packets from Sony PC app and compare to phone output before writing LUMM bridge. (2) Flat-packed fallback: mocopi-bridge.py already tries 29-byte-per-bone flat parsing as Strategy 1. (3) Phone fallback: if PC app parsing fails, use phone app -> OSC -> mocopi_bridge.py OSC listener.
- Validation: mocopi_bridge.py --verbose mode prints parsed joint positions. Visually verify root/hips move when performer walks.

LOW RISKS

R8: NSSM Service PATH Issues
- Probability: 15
- Mitigation: Use absolute python.exe path in NSSM config. Test on K11 fresh login.

R9: Firewall Blocks UDP on K11
- Probability: 40
- Mitigation: firewall-rules.ps1 runs on Day 1.

R10: Code Drift Between Mac Dev and K11 Deploy
- Probability: 30
- Mitigation: Same git repo, same branch. rsync or git pull on K11.

3b. Expanded Specifications

Spec 1: mocopi_bridge.py (LUMM Publisher)

Purpose: Receive Sony mocopi binary data on :12351 (or OSC on :9500), parse 27-bone skeleton, publish as LUMM :9702.

Input sources (priority order):
1. Sony mocopi PC app on K11 -> :12351 UDP (primary on K11)
2. Sony mocopi phone app -> :9500 OSC (fallback, phone in pocket)
3. Synthetic: generate walking/dancing skeleton for testing

LUMM wire format (from K11 production plan):

Header (16 bytes):
  u32 magic       = 0x4C554D4D
  u32 frame       = monotonic counter
  u16 bone_count  = 27
  u16 flags       = bit0: has_quat, bit1: has_pos, bit2: is_calibrated
  f32 dt_seconds  = wall-time delta

Body (756 bytes = 27 * 28):
  Per bone (28 bytes):
    f32 position[3]    = x, y, z (parent-relative)
    f32 quaternion[4]  = x, y, z, w (local rotation)

Total: 772 bytes fixed, sub-MTU, no chunking.

Sony TLV parsing (from existing mocopi-bridge.py):
- Magic: "MCS\0" header or TLV 'sndf' -> 'fram' -> 'btdt' nesting
- Bone data: TLV 'btdt' -> 'bnid' (uint16) + 'tran' (7 floats: qx,qy,qz,qw,px,py,pz)
- Flat fallback: 29 bytes/bone (bone_id u8 + quat 16B + pos 12B)

OSC fallback (from existing mocopi_receiver.py):
- Listen :9500 for /mocopi/bone/* addresses
- Parse float arrays for position + quaternion

CLI:

python3 mocopi_bridge.py --host [ip] --port 9702 --sony-port 12351 --osc-port 9500 --synthetic --fps 50

Behavioral requirements:
- Prefer Sony source when both are active (Sony has lower latency)
- Publish even when no sensors connected (zero-filled bones, flags=0x0000)
- Log active bone count every 5 seconds
- --verbose mode prints parsed joint positions

Spec 2: LumeMocopiReceiver.cs (Unity LUMM Consumer)

Purpose: Receive LUMM :9702, parse 27-bone skeleton, expose as shader globals and C# data.

Implementation:

[DefaultExecutionOrder(100)]
public class LumeMocopiReceiver : MonoBehaviour
{
    public int listenPort = 9702;

    // Shader globals
    // _MocopiBonePositions: float4[27] (xyz + bone_id in w)
    // _MocopiRootPosition: float4 (hips world position)
    // _MocopiRootVelocity: float (magnitude of root position delta)
    // _MocopiFlags: int (calibrated, has_quat, has_pos)
    // _MocopiBoneCount: int (active bones, 0 if stale)

    // Public C# access
    public Vector3[] BonePositions { get; }
    public Quaternion[] BoneRotations { get; }
    public float RootVelocity { get; }
    public bool IsCalibrated { get; }
    public bool IsStale { get; } // true if no packet for >200ms
}

Stale handling: After 200ms without a LUMM packet, set _MocopiBoneCount=0 and IsStale=true. Consumers must check this before using bone data.

Spec 3: LumeSystemBus.cs (LUMS Publisher)

Purpose: Aggregate K11 scene state, publish as LUMS :9703 for remote consumers.

LUMS wire format (new):

Magic: 0x4C554D53 ('LUMS'), Port: :9703

Header (16 bytes):
  u32 magic         = 0x4C554D53
  u32 frame         = monotonic
  u16 field_count   = 12 (v1)
  u16 flags         = reserved
  f32 dt_seconds    = wall-time delta

Body (48 bytes = 12 * 4):
  f32 motion_energy     = _InnerSpread (from LumeOpticalFlow)
  f32 flow_magnitude    = flow field RMS (from LumeOpticalFlow)
  f32 time_scale        = _LumeTimeScale (from LumeMotionGate)
  f32 audio_rms         = _AudioLevels.x (from LumeAudioFftReceiver)
  f32 audio_band_sub    = _AudioLevels.y (sub band)
  f32 audio_band_low    = _AudioLevels.z (low band)
  f32 audio_band_mid    = _AudioLevels.w (mid band)
  f32 transient_active  = 1.0 if transient this frame, else 0.0
  f32 bone_count        = _MocopiBoneCount (from LumeMocopiReceiver)
  f32 root_velocity     = _MocopiRootVelocity
  f32 preset_id         = current VFX preset index
  f32 frame_rate        = 1.0 / deltaTime

Total: 64 bytes fixed, sub-MTU, no chunking. Published at 30Hz.

Spec 4: echelon_relay.py (Control-to-Data Plane Bridge)

Purpose: Subscribe to multicam-server :9404/director/ws, extract echelon events and bar-fire events, publish as LUME_ECH :9704 for Unity consumption.

LUME_ECH wire format (new):

Magic: 0x4C554D48 ('LUMH' -- echelon/harmony), Port: :9704

Header+Body (48 bytes fixed):
  u32 magic           = 0x4C554D48
  u32 frame           = monotonic
  u8  section_state   = 0-6
  u8  scoring_mode    = 0=idle, 1=performance, 2=creator, 3=photoshoot
  u16 flags           = bit0: bar_boundary_now
  f32 tension         = 0.0-1.0
  f32 divergence      = 0.0-1.0
  f32 soft_tempo_bpm  = estimated BPM
  f32 bar_phase       = 0.0-1.0 (position within current bar)
  f32 energy_arc      = 0.0-1.0 (overall performance energy)
  f32 dt_seconds      = wall-time delta
  u32 bar_count       = total bars since session start
  u32 reserved[2]     = 0

Behavioral requirements:
- Connect to ws://{mac1_ip}:9404/director/ws on startup
- Parse JSON messages for "echelon" and "bar_fire" event types
- Publish LUME_ECH at 10Hz (every 100ms)
- On bar_fire event: set bar_boundary_now flag for one frame
- Auto-reconnect on disconnect (1s, 2s, 4s backoff, max 10s)
- --host flag for Unity listener address (default [ip])

Spec 5: iOS Receiver Trio (LummReceiver + LumsReceiver + LumeAudioReceiver)

LummReceiver.swift (~80 lines):
- Bind UDP :9702, parse LUMM 772B packets
- Extract 27 bone positions + quaternions
- Feed MocopiFeatureExtractor on receipt
- Stale timeout: 3s -> clear feature buffer, zero 128D[76:100]
- Background UdpClient pattern (matches LumeAudioFftReceiver.cs approach)

LumsReceiver.swift (~60 lines):
- Bind UDP :9703, parse LUMS 64B packets
- Extract scene state: motion_energy, flow_magnitude, time_scale, audio features
- Feed into 128D[108:116]: motion_energy, flow_magnitude, time_scale, bone_count (4 floats)
- Additional signals available for SAN via a published struct

LumeAudioReceiver.swift (~40 lines):
- Bind UDP :9701, parse LUMF 84B packets
- Extract RMS, spectral centroid, onset, 4 bands
- Feed into 128D[104:108]: rms, centroid_normalized, onset, dominant_band
- Provides ambient audio awareness to SAN (room mic hears what the audience hears)

3c. Master Execution Checklist

### Wave 0: K11 Bootstrap (Day 1)
| # | Task | Input | Output | Validation | Owner | Status | automate |
|---|------|-------|--------|------------|-------|--------|----------|
| 1 | K11 first-boot, Windows setup | K11 hardware | Running Windows 11 | Desktop visible | Mohamed | TODO | false |
| 2 | Install Git, Python 3.12, pip | Windows 11 | git, python3, pip in PATH | `python --version` | Mohamed | TODO | true |
| 3 | Install Tailscale, join mesh | Windows 11 | K11 visible in Tailscale admin | `ping mac1` succeeds | Mohamed | TODO | false |
| 4 | Git clone lume-commerce to C:\Users\lume\lume | Git | Repo on K11 | `ls software/demo/` | Mohamed | TODO | true |
| 5 | pip install numpy sounddevice | Python | Dependencies ready | `python -c "import numpy"` | Mohamed | TODO | true |
| 6 | Run pointcloud_pub.py --synthetic | Repo | LUME :9700 publishing | lume_packet_inspector.py shows frames | Agent | TODO | true |
| 7 | Run audio_pub.py --synthetic | Repo | LUMF :9701 publishing | inspector shows 60Hz | Agent | TODO | true |
| 8 | Verify UDP from Mac1 via Tailscale | Tailscale | Packets cross mesh | inspector on Mac1 sees K11 frames | Agent | TODO | true |
| 9 | Run firewall-rules.ps1 | Script | UDP ports allowed | No firewall prompts on publisher start | Agent | TODO | true |

Kill criterion: K11 can't run Python, or Tailscale can't reach K11 from Mac1.

### Wave 1: Femto Bolt + UMA-8 (Day 2)
| # | Task | Input | Output | Validation | Owner | Status | automate |
|---|------|-------|--------|------------|-------|--------|----------|
| 10 | Connect Femto Bolt USB-C | Hardware | USB device enumerated | `lsusb` equivalent / Device Manager | Mohamed | TODO | false |
| 11 | Install Orbbec Viewer, verify depth | Femto Bolt | Depth preview visible | 640x576 ToF image | Mohamed | TODO | false |
| 12 | Install pyorbbecsdk (pip) | Python | SDK importable | `python -c "from pyorbbecsdk import Pipeline"` | Agent | TODO | true |
| 13 | Run pointcloud_pub.py --raw-depth | Femto Bolt | Real LUMD :9700 | inspector shows LUMD magic, 640x576 | Agent | TODO | true |
| 14 | Connect UMA-8 USB-A | Hardware | Audio device enumerated | `sounddevice.query_devices()` lists UMA-8 | Mohamed | TODO | false |
| 15 | Run audio_pub.py --mic --channels 8 | UMA-8 | Real LUMF :9701 | inspector shows RMS > 0 with ambient sound | Agent | TODO | true |
| 16 | Caliper-verify K11 + Bolt dimensions | Hardware | Confirmed measurements | Matches CAD within 1mm | Mohamed | TODO | false |

Kill criterion: pyorbbecsdk can't enumerate Bolt on Windows.

### Wave 2: Unity on K11 (Day 2-3)
| # | Task | Input | Output | Validation | Owner | Status | automate |
|---|------|-------|--------|------------|-------|--------|----------|
| 17 | Install Unity Hub + 6000.3.2f1 | Windows | Unity installed | Unity Editor launches | Mohamed | TODO | false |
| 18 | Open lume_pcloud project, resolve packages | Unity project | Project compiles | 0 errors in Console | Mohamed | TODO | false |
| 19 | Build Player (Windows x86_64, Vulkan) | Unity project | lume_pcloud.exe | .exe in Builds/Windows/ | Agent | TODO | true |
| 20 | Run with --synthetic publishers | lume_pcloud.exe + synthetic | Visual output on screen | Torus + depth palette visible | Agent | TODO | true |
| 21 | Run with real Bolt depth | lume_pcloud.exe + Bolt | Depth-reactive visuals | Moving in front of camera changes scene | Mohamed | TODO | false |
| 22 | Run with real UMA-8 audio | lume_pcloud.exe + UMA-8 | Audio-reactive visuals | Clap triggers outline flash | Mohamed | TODO | false |
| 23 | Profile frame time on Radeon 780M | Unity Profiler | Frame time measurement | <12ms average (60fps headroom) | Agent | TODO | true |

Kill criterion: Compute shaders fail on Radeon 780M.

### Wave 3: LUMM mocopi Bridge (Day 3-4, parallel with Wave 4)
| # | Task | Input | Output | Validation | Owner | Status | automate |
|---|------|-------|--------|------------|-------|--------|----------|
| 24 | Write LUMM wire format spec in lume_wire_format.md | K11 plan spec | Updated doc | Spec matches 772B layout | Agent | TODO | true |
| 25 | Write mocopi_bridge.py (LUMM publisher) | Sony protocol + LUMM spec | ~200 lines Python | `python mocopi_bridge.py --synthetic` publishes at 50Hz | Agent | TODO | true |
| 26 | Write test_mocopi_bridge.py | Bridge code | ~150 lines tests | pytest green: header parse, bone round-trip, synthetic | Agent | TODO | true |
| 27 | Update lume_packet_inspector.py | LUMM magic | +LUMM +LUMS +LUMH magics | inspector recognizes LUMM frames | Agent | TODO | true |
| 28 | Test with Sony mocopi PC app on K11 | Sony app + 6 sensors | LUMM from real skeleton | inspector shows bone_count=27 | Mohamed | TODO | false |
| 29 | Test phone fallback (OSC :9500 path) | Phone mocopi app | LUMM from OSC source | inspector shows frames from OSC path | Mohamed | TODO | false |

Kill criterion: Sony binary protocol parsing fails AND phone OSC fallback fails.

### Wave 4: Unity LUMM + LUMS + ECH (Day 4-5)
| # | Task | Input | Output | Validation | Owner | Status | automate |
|---|------|-------|--------|------------|-------|--------|----------|
| 30 | Write LumeMocopiReceiver.cs | LUMM spec | ~120 lines C# | Receives LUMM, exposes BonePositions[] in Inspector | Agent | TODO | true |
| 31 | Write LumeSystemBus.cs (LUMS publisher) | LUMS spec | ~100 lines C# | inspector on Mac1 sees LUMS 64B at 30Hz from K11 | Agent | TODO | true |
| 32 | Write LUMS wire format spec | Step 6 design | Updated doc | Spec in lume_wire_format.md | Agent | TODO | true |
| 33 | Write echelon_relay.py | LUMH spec + WS | ~80 lines Python | Connects to :9404/director/ws, publishes LUMH :9704 | Agent | TODO | true |
| 34 | Write LumeEchelonReceiver.cs | LUMH spec | ~80 lines C# | Receives LUMH, exposes section_state in Inspector | Agent | TODO | true |
| 35 | Write LUMH wire format spec | Step 5 design | Updated doc | Spec in lume_wire_format.md | Agent | TODO | true |
| 36 | Update LumeWaveAutoWire.cs | 3 new components | Auto-wire attaches all 13 components | "Lume/Auto-Wire" menu works | Agent | TODO | true |
| 37 | Update LumeCalibrationPanel.cs | 3 new components | F12 panel shows mocopi/echelon tunables | F12 shows bone_count, section_state | Agent | TODO | true |
| 38 | Wire mocopi bones to shader globals | LumeMocopiReceiver | _MocopiBonePositions visible in shader | Skeleton data in frame debugger | Agent | TODO | true |
| 39 | Rebuild Unity Player | All Wave 4 code | Updated lume_pcloud.exe | .exe runs with all new components | Agent | TODO | true |

### Wave 5: MotionMix iOS Integration (Day 5)
| # | Task | Input | Output | Validation | Owner | Status | automate |
|---|------|-------|--------|------------|-------|--------|----------|
| 40 | Write LummReceiver.swift | LUMM spec | ~80 lines Swift | Receives LUMM, feeds MocopiFeatureExtractor | Agent | TODO | true |
| 41 | Write LumsReceiver.swift | LUMS spec | ~60 lines Swift | Receives LUMS, feeds 128D[108:116] | Agent | TODO | true |
| 42 | Write LumeAudioReceiver.swift | LUMF spec | ~40 lines Swift | Receives LUMF, feeds 128D[104:108] | Agent | TODO | true |
| 43 | Wire receivers into EchelonBridge.swift | 3 receivers | Updated 128D assembly | 128D[76:116] populated when K11 publishing | Agent | TODO | true |
| 44 | Wire receivers into MotionMixApp.swift | 3 receivers | Receivers initialized on launch | No crash on startup, receivers bind ports | Agent | TODO | true |
| 45 | Add settings toggles for LUME receivers | SettingsView.swift | UI toggles | Can enable/disable each receiver | Agent | TODO | true |
| 46 | End-to-end: K11 publishes -> iOS receives | Running system | 128D enriched with K11 data | Debug print shows [76:116] non-zero | Mohamed | TODO | false |

### Wave 6: NSSM Services + Production (Day 5-6)
| # | Task | Input | Output | Validation | Owner | Status | automate |
|---|------|-------|--------|------------|-------|--------|----------|
| 47 | Write install-services.ps1 | NSSM + scripts | 3 NSSM services registered | `nssm status lume-depth` shows Running | Agent | TODO | true |
| 48 | Write health-check.ps1 | Windows env | Diagnostic tool | Shows PID + port status for all services | Agent | TODO | true |
| 49 | Register Unity in Task Scheduler | Task Scheduler | Auto-launch on login | lume_pcloud.exe starts after login | Mohamed | TODO | false |
| 50 | Promote mocopi_bridge.py to NSSM | NSSM | 4th NSSM service | `nssm status lume-mocopi` shows Running | Agent | TODO | true |
| 51 | Write lume-k11.json config | All port/host values | Config file | All hardcoded values in one place | Agent | TODO | true |

### Wave 7: Integration Testing + Hardening (Day 6-7)
| # | Task | Input | Output | Validation | Owner | Status | automate |
|---|------|-------|--------|------------|-------|--------|----------|
| 52 | 30-min sustained load test | Full system running | No crashes or memory leaks | All services still running after 30 min | Mohamed | TODO | false |
| 53 | Cold-boot end-to-end dry run | K11 power cycle | Full system auto-starts | Visual output within 60s of login | Mohamed | TODO | false |
| 54 | Measure latency budget (visual path) | Camera + display | Frame-to-pixel measurement | <20ms | Mohamed | TODO | false |
| 55 | Measure latency budget (audio path) | Mocopi -> audio output | Skeleton-to-sound measurement | <80ms | Mohamed | TODO | false |
| 56 | Test Tailscale failover | Disconnect Tailscale | Graceful degradation | K11 visuals continue, iOS falls back to local | Mohamed | TODO | false |
| 57 | Update ARCHITECTURE.md | All new code | Updated architecture doc | Matches reality | Agent | TODO | true |
| 58 | Write system runbook | Operating experience | Runbook document | Covers startup, shutdown, troubleshooting | Agent | TODO | true |

Summary

Dependency Chain

Wave 0 (bootstrap)
  |
  +-> Wave 1 (sensors) -- needs K11 running
  |     |
  |     +-> Wave 2 (Unity) -- needs sensors verified
  |           |
  |           +-> Wave 3 (LUMM bridge) ----+
  |           |                             |
  |           +-> Wave 4 (Unity new code) --+-- needs bridge + Unity
  |                 |
  |                 +-> Wave 5 (iOS) -- needs LUMM/LUMS/LUMH publishing
  |                       |
  |                       +-> Wave 6 (NSSM) -- needs everything working
  |                             |
  |                             +-> Wave 7 (hardening) -- needs production deploy
  |
  Wave 3 and Wave 4 can partially overlap (bridge before Unity consumers)