Stage 0: RESEARCH -- LUME Full System Architecture

Stage 0: RESEARCH -- LUME Full System Architecture

What Exists Today

### Hardware Stack
- K11 (GMKtec NucBox K11): Ryzen 9 8945HS, Radeon 780M (~9 TFLOPS), 32GB DDR5, 1TB NVMe. Windows 11 Pro. No CUDA.
- Orbbec Femto Bolt: USB-C depth camera, 640x576 ToF (confirmed via pyorbbecsdk, published LUMD on :9700)
- Orbbec Femto Mega: Legacy camera, larger with rear Jetson hump. USB-C.
- UMA-8 mic array: 8-channel USB-A. Publishes LUMF on :9701 via audio_pub.py.
- Sony mocopi: 6x BLE IMU sensors. Native app on phone publishes to :12351. LUMM wire format spec exists but bridge not yet deployed.
- Network: Tailscale mesh connecting mac1, mac4, mac5, cloud-vm, K11.

### LUME Software on K11 (existing, verified)
Python publishers (3 files, ~1170 lines total):
- `pointcloud_pub.py` (556 lines) -- LUME/LUMD dual-format. Real Femto via pyorbbecsdk or synthetic. Ships 40B LUMD headers with intrinsics in every datagram.
- `audio_pub.py` (315 lines) -- LUMF 84B fixed datagrams. RMS, 4 EQ bands, 8 MFCC, onset/transient detection. Mic mode via sounddevice or synthetic.
- `mocopi_bridge.py` (256 lines, at MotionMix/server/) -- Parses Sony binary TLV protocol from :12351, extracts 27-bone positions. Currently forwards as JSON to capture server HTTP :9405. Needs rewrite to publish LUMM UDP.

Unity project (22 C# files, ~3200 lines):
- `LumeUdpReceiver.cs` -- Magic-byte dispatch for LUME/LUMD
- `LumePointRenderer.cs` -- Cloud/Depth render modes, GraphicsBuffer
- `LumeDepthReprojector.cs` -- GPU pinhole reprojection from LUMD
- `LumeOpticalFlow.cs` -- Frame-diff scalar + Lucas-Kanade dense flow
- `LumeAudioFftReceiver.cs` -- LUMF consumer, [DefaultExecutionOrder(200)]
- `LumeVfxRuntimeBridge.cs` -- Pushes globals to VFX Graph, [DefaultExecutionOrder(210)]
- `LumeTransientForcePusher.cs` -- Impulse events from transients, [DefaultExecutionOrder(220)]
- `LumeCalibrationPanel.cs` -- F12 IMGUI panel, JSON persistence
- `LumeMotionGate.cs` -- SuperHot-style time scaling from motion, [DefaultExecutionOrder(150)]
- `LumeProceduralTunnel.cs` -- 6144-point cylindrical helix
- `LumeAudioReactor.cs` -- Outline flash + HSV jitter from audio
- `LumeCalibration.cs` -- Calibration storage
- `LumeVfxEditor.cs` + 5 preset ScriptableObjects -- VFX presets
- 3 Editor scripts: AutoWire, BootstrapTunnelScene, VfxGraphBootstrap
- 2 Compute shaders: LumeDepthReproject.compute, LumeOpticalFlow.compute

Wire formats (documented in lume_wire_format.md):
| Magic | Format | Port | Size | Status |
|-------|--------|------|------|--------|
| 0x4C554D45 LUME | Synthetic cloud | :9700 | Variable (450pt chunks) | Working |
| 0x4C554D44 LUMD | Raw depth | :9700 | 40B hdr + 716px chunks | Working |
| 0x4C554D46 LUMF | Audio FFT | :9701 | 84B fixed | Working |
| 0x4C554D4D LUMM | Mocopi skeletal | :9702 | 772B fixed (27 bones) | Spec exists, bridge not deployed |
| 0x4C554D43 LUMC | Color-fused | :9700 | Reserved, not implemented | Spec only |

Test coverage: 37 Python tests (33 fast + 4 slow), including wire format golden tests.

### MotionMix iOS App (existing, verified)
Core services:
- `EchelonBridge.swift` (~250+ lines) -- 60Hz main loop on @MainActor. Owns SANService. Rust FFI to Echelon engine via libechelon_ios.a. Processes CoreMotion sensor data, computes 128D canonical dynamics vector.
- `ParamMapper.swift` (218 lines) -- Maps continuous body motion to audio parameters. Right arm -> brightness/pads, left arm -> density/percussion. Velocity-boosted. Presence detection with snap-back decay. Cadence detection, rhythm stability, motion accent.
- `ChestFlexDetector.swift` (330 lines) -- Vision BodyPose landmark analysis. Shoulder width velocity + torso bounce -> flex detection. L/R asymmetry, combo system.
- `StrudelWebEngine.swift` (236 lines) -- Hidden WKWebView running Strudel.js from Mac5 :9600. Controls: play(pattern), setTempo(bpm), stop(). SAN pusher at 10Hz to local webview + 2Hz POST to Mac5 /san endpoint.
- `AudioEngine.swift` (~80+ lines visible) -- AVAudioEngine with Echelon Rust rendering OR Swift fallback. Source nodes for real-time synthesis.
- `LiveStreamServer.swift` (392 lines) -- MJPEG stream server on :8081. Token-authenticated. Preview (10fps) and Program (18fps) profiles.
- `MocopiFeatureExtractor` -- Compresses 27-bone skeleton to 24D features at [76:100] in 128D canonical vector.

128D Canonical Vector Layout:

[0:32]    z vector (16D LIM-RPS latent, padded)
[32:64]   velocity (16D, padded)
[63:69]   pose features (Mocopi or Vision)
[69:75]   temporal scalars (tempo, phase, periodicity, grounding, verticality, rotation, coherence)
[75]      modality mask (cam=1, pocket=2, mocopi=4, watch=8)
[76:100]  Mocopi 24D features
[100:102] Pocket IMU (pitch, roll)
[102:104] Watch (HR normalized, wrist energy)
[104:128] reserved (future: Femto Bolt, LiDAR, etc.)

### Multicam Server (existing, Rust/Axum, verified)
- `main.rs` (10,678 lines) -- Binds [ip]:9404. 70+ routes.
- Director loop runs at 250ms tick. Murch V2 scoring for performance mode, photoshoot scoring for stills.
- EchelonState tracked: phase, soft_tempo_bpm, bar boundaries, bar_count.
- BeatQuantizer for beat-quantized cuts.
- TemporalIntelligence for adaptive cut timing.
- Mocopi joint normalization already implemented (canonicalize_joint_name, normalize_joint_payload).
- WebSocket hubs: /director/ws, /device/ws, /td/ws, /hero-feed/ws.
- Session management: start, status, presets, autonomous, manifest.
- Stills pipeline: capture, upload, star, proof links, lookbook rendering.

### Echelon Engine (existing, Rust, 20 crates)
- `cc-echelon/crates/` contains: audio-engine, cc-brain, control-bus, dell, dsp-utils, echelon-ios, link-clock, media, midi-osc, motion-bridge, music-brain, phrase-intelligence, scheduler, shmem-ipc, ts-shift, ui-shell, viz, viz-server, voice-control
- `cc-brain/src/` contains: anticipation_adapter.rs, conductor.rs, ffi.rs, san/ (SAN pipeline), state_machine/, latent/, lexicon/, nn/, types.rs
- Compiled to libechelon_ios.a for iPhone (arm64).
- SAN (Somatic Adaptive Network): FAN -> FuseMoE -> NHA -> TTT -> FiLM heads. 135K params.
- link-clock: Ableton Link FFI wrapper for synchronized beat clocks.
- Not compiled for Windows/x86_64 currently.

### Computational Choreography Libraries
- `comp-core-data/` and related directories at Desktop/Comp-Core/
- Gesture recognition, movement quality analysis, motion-to-music parameter mapping
- The 128D canonical vector IS the handoff format between motion input and music output

### Diffusion/ML Inference
- No CUDA on K11 (Radeon 780M = AMD, no CUDA support)
- One-step diffusion models exist in projects/ but require CUDA
- Must run on Mac5 (M-series Apple Silicon with MLX) or cloud-vm (GPU)
- ConditioningEncoder.mlpackage on disk is 104D input (needs retrain to 128D)

What's Been Tried Before

### K11 Production Plan (2026-04-27 Evo3)
- Three-tier hybrid: NSSM services + Task Scheduler + Sony native
- 8 waves, 52 tasks, 5-7 day critical path
- LUMM wire format spec finalized: 772B, 27 bones
- Key risk: compute shaders on Radeon 780M Vulkan (15

### LUME Wave 1-5 Build History
- Wave 1: Synthetic cloud + audio (Mac5 deploy, LaunchAgents)
- Wave 2: Raw depth reprojection (GPU pinhole)
- Wave 3: Optical flow (frame-diff scalar)
- Wave 4: LK dense flow, LUMF audio sidecar, VFX bootstrap, wire format tests, auto-wire, integration tests
- Wave 5: Transient impulse, calibration UI, SuperHot motion gate

### MotionMix Unified Director Evo3 (2026-04-04)
- Identified dual-director conflict (iOS AutoDirector at 60Hz vs server director_loop at 250ms)
- Solution: server-sovereign directing, iOS as display-only
- Creator mode (face+speech) vs Performance mode (Echelon+beat)

Real Constraints

1. K11 has no CUDA. Any ML inference requiring CUDA must be offloaded to Mac5 or cloud-vm via Tailscale API.
2. K11 runs Windows 11. Python publishers and Unity work, but no LaunchAgents (NSSM services instead).
3. Echelon is iOS-only compiled. libechelon_ios.a is arm64. The Rust source exists but has never been compiled for Windows x86_64.
4. Single operator. Mohamed is the sole developer, performer, hardware assembler, and content creator.
5. Latency budget. Live performance requires sub-50ms sensor-to-visual. UDP is mandatory for real-time paths.
6. Wire format lock. LUME/LUMD/LUMF/LUMM are frozen and tested. Adding new formats is fine; changing existing ones breaks deployed consumers.
7. Unity 6000.3.2f1 + URP 17 + VFX Graph 17. Compute shaders must work on Vulkan (primary) or DX12 (fallback) on Radeon 780M.
8. Strudel.js runs on Mac5 :9600. The audio synthesis engine is NOT on K11.
9. Multicam server runs on Mac1 :9404. It is the coordination hub for all iOS cameras and the director.

Open Questions

1. Should Echelon be compiled for Windows x86_64 on K11, or should it remain iOS-only with MotionMix as the brain?
2. Where does music synthesis happen? K11 (new audio engine)? Mac5 (Strudel.js)? MotionMix iOS (existing AudioEngine)?
3. How do mocopi skeletal features reach the 128D canonical vector? Via LUMM -> MotionMix iOS -> MocopiFeatureExtractor? Or via LUMM -> K11 local feature extraction?
4. What is the role of the multicam server :9404 in the LUME system? Does it remain the coordination hub or does a new hub emerge?
5. How does Echelon's beat clock synchronize across K11 (visuals) and Mac5 (audio)?
6. Should the LUME packet inspector become a health monitoring service?
7. What happens when Tailscale mesh has packet loss (documented: Mac5 unreachable during some sessions)?