Stage 2: COMPOUND -- Sequential Synthesis
| Conflict | Resolution | Reasoning | |----------|-----------|-----------| | Who owns visual reactivity? | K11 Unity, locally (Path A/F win over C) | Visual latency budget is ~13ms. Remote control adds 20-40ms round trip. Unacceptable. | | Who owns music intelligence? | MotionMix iOS (Path C wins over A/D) | EchelonBridge + SAN + ParamMapper is 5,000+ lines of battle-tested motion-to-music. No port. | | Who coordinates across mesh? | Multicam server :9404 on Mac1 (Path B insight, simplified) | Already exists, alrea
Full Public Reader
Stage 2: COMPOUND -- Sequential Synthesis
Resolution of Path Conflicts
| Conflict | Resolution | Reasoning |
|---|---|---|
| Who owns visual reactivity? | K11 Unity, locally (Path A/F win over C) | Visual latency budget is ~13ms. Remote control adds 20-40ms round trip. Unacceptable. |
| Who owns music intelligence? | MotionMix iOS (Path C wins over A/D) | EchelonBridge + SAN + ParamMapper is 5,000+ lines of battle-tested motion-to-music. No port. |
| Who coordinates across mesh? | Multicam server :9404 on Mac1 (Path B insight, simplified) | Already exists, already has director loop + WebSocket hubs + session management. |
| Message bus for sensor data? | Direct UDP, no NATS (Path F wins over E) | Real-time sensor data needs <2ms local, not 15ms through a broker. |
| Message bus for control plane? | HTTP/WebSocket via multicam server (Path E insight, existing infra) | :9404 already handles /echelon, /bar-fire, /director/ws, /session/*. |
| Does Unity publish outward? | Yes: LUMS derived state (Path D insight) | Unity publishes scene state so remote consumers get processed features, not raw data. |
| Where does ML inference run? | cloud-vm or Mac5, REST API via Tailscale (Path B) | K11 has no CUDA. One-step diffusion stays remote. |
Compound Step 1: Ground Truth -- The Two-Plane Architecture
Start from scratch. The LUME system has exactly two communication planes:
Data Plane (real-time, UDP, sub-50ms):
All sensor data and derived features flow as UDP datagrams using the LUME wire format family. Publishers bind [ip]. Consumers listen on well-known ports. No middleware. No broker. No acknowledgment. Fire-and-forget with graceful degradation on loss.
Control Plane (coordination, HTTP/WS, 100ms+ tolerance):
Session management, director cuts, echelon state sync, health checks, and configuration flow through the multicam server :9404 HTTP API and WebSocket hubs. Reliable delivery. Request-response for commands. Event streams for state changes.
This is not a new idea. It is an explicit naming of what already exists.
+=====================================================+
| DATA PLANE (UDP) |
| LUMD :9700 | LUMF :9701 | LUMM :9702 | LUMS :9703 |
| (depth) | (audio) | (skeleton) | (scene state)|
| <2ms local | <2ms local | <2ms local | 30Hz outward |
+=====================================================+
+=====================================================+
| CONTROL PLANE (HTTP/WS) |
| multicam-server :9404 |
| /echelon | /bar-fire | /director/ws | /session/* |
| /health | /devices | /td/ws | /mesh/* |
| 100ms+ OK | reliable | persistent | coordinated |
+=====================================================+Compound Step 2: The Three Sovereignty Domains
Building on Step 1's two-plane architecture, assign clear sovereignty:
Domain 1: K11 -- Visual Sovereignty
K11 owns everything between sensor input and HDMI output. It runs the Python publishers, Unity scene, and all visual processing components. It does NOT make music decisions. It does NOT coordinate other machines. It publishes derived visual state (LUMS) for remote consumers.
Domain 2: MotionMix iOS -- Musical Sovereignty
The iPhone owns the motion-to-music pipeline. It consumes LUMM (skeleton) and LUMF (audio environment) from K11 via Tailscale UDP, fuses them with local CoreMotion/Vision data into the 128D canonical vector, runs the SAN pipeline, and drives Strudel.js audio synthesis on Mac5. It does NOT control K11 visuals directly.
Domain 3: Multicam Server Mac1 -- Coordination Sovereignty
The multicam server owns session lifecycle, director cuts, beat quantization, and cross-device state. It does NOT process sensor data. It does NOT render anything. It receives state updates from both K11 and MotionMix and makes coordination decisions.
+-------------------+ +-------------------+ +-------------------+
| K11 (Visual) | | iOS (Musical) | | Mac1 (Coord) |
| | | | | |
| Sensors -> Unity | | 128D Canonical | | Director Loop |
| -> HDMI Display | | SAN -> Strudel | | Session Mgmt |
| -> LUMS :9703 | | -> Mac5 Audio | | Beat Quantizer |
| | | | | |
| DOES NOT: music | | DOES NOT: visuals | | DOES NOT: render |
| DOES NOT: coord | | DOES NOT: coord | | DOES NOT: sensors |
+-------------------+ +-------------------+ +-------------------+Compound Step 3: Complete Data Flow -- Sensor to Output
Building on Steps 1-2, trace every data path:
Path A: Depth Camera -> Visual Output (K11 local, ~13ms)
Femto Bolt (USB-C, 640x576 ToF, 30fps)
-> pyorbbecsdk (in pointcloud_pub.py)
-> LUMD :9700 (UDP loopback, 40B hdr, 716px chunks)
-> LumeUdpReceiver (magic dispatch LUMD)
-> LumeDepthReprojector (GPU pinhole, compute shader)
-> LumeOpticalFlow (frame-diff scalar + LK dense flow)
-> LumeMotionGate (motion -> _LumeTimeScale global)
-> LumePointRenderer (Cloud/Depth mode, instanced draw)
-> LumeAudioReactor (outline flash from audio)
-> LumeVfxRuntimeBridge -> VFX Graph particles
-> URP 17 render pipeline -> HDMI outPath B: Microphone -> Visual Audio Reactivity (K11 local, ~8ms)
UMA-8 (USB-A, 8ch, 48kHz)
-> sounddevice (in audio_pub.py --mic)
-> LUMF :9701 (UDP loopback, 84B fixed, 60fps)
-> LumeAudioFftReceiver (RMS, 4 bands, transient bit)
-> _AudioLevels, _OutlineFlash, _InnerSpread globals
-> LumeAudioReactor (HSV jitter, outline flash)
-> LumeTransientForcePusher (impulse events on transient)
-> LumeVfxRuntimeBridge -> VFX GraphPath C: Microphone -> Music Intelligence (K11 -> iOS, ~25ms)
audio_pub.py publishes LUMF to [ip]:9701
-> Tailscale UDP (~10ms) -> MotionMix iOS
-> LumeAudioReceiver.swift (NEW, ~40 lines)
-> Ambient audio features -> 128D[104:108]
-> EchelonBridge.step() -> SAN inference
-> ParamMapper.update() -> music params
-> StrudelWebEngine -> Mac5 Strudel.js :9600Path D: Mocopi Skeleton -> Visual Output (K11 local, ~20ms)
Sony mocopi (6x BLE IMU, ~50Hz)
-> Sony mocopi PC app (K11 Windows)
-> :12351 (localhost, Sony binary TLV)
-> mocopi_bridge.py (parse + reformat)
-> LUMM :9702 (UDP loopback, 772B fixed)
-> LumeMocopiReceiver.cs (NEW, in Unity)
-> 27 bone positions + quaternions
-> Shader globals: _MocopiBonePositions[27], _MocopiRootVelocity
-> Visual effects: skeleton-reactive particles, silhouettePath E: Mocopi Skeleton -> Music Synthesis (K11 -> iOS -> Mac5, ~50ms)
mocopi_bridge.py publishes LUMM to [ip]:9702
-> Tailscale UDP (~10ms) -> MotionMix iOS
-> LummReceiver.swift (NEW, 80 lines, from K11 plan)
-> MocopiFeatureExtractor -> 24D features
-> 128D[76:100] (joint velocities, limb ratios)
-> EchelonBridge.step() -> SAN inference
-> ParamMapper.update() -> music params
-> StrudelWebEngine -> Mac5 Strudel.js :9600 (~10ms)
-> Strudel audio render (~8ms) -> speakersPath F: Phone Sensors -> Music (iOS local, ~5ms)
iPhone CoreMotion (accelerometer, gyro, attitude, gravity)
-> EchelonBridge.updateSensor() (queued, drained at 60Hz)
-> echelon_update_sensor() (Rust FFI)
-> echelon_step() -> 16D LIM-RPS latent
-> 128D[0:75] (z vector, velocity, temporal scalars)
iPhone Vision BodyPose (from camera)
-> cachedPoseFeatures -> 128D[63:69]
-> SAN.step() -> music params
-> ParamMapper -> StrudelWebEnginePath G: K11 Scene State -> MotionMix (K11 -> iOS, ~15ms) [NEW]
Unity LumeSystemBus.cs (NEW, [DefaultExecutionOrder(250)])
-> Aggregates: motion_energy, flow_magnitude, time_scale,
audio_rms, transient_active, preset_id, bone_count
-> LUMS :9703 (0x4C554D53, ~64B fixed, 30Hz)
-> Tailscale UDP -> MotionMix iOS
-> LumsReceiver.swift (NEW)
-> Visual-scene awareness for music decisions
-> e.g., when flow_magnitude spikes (dancer enters frame),
increase music energy. When time_scale drops (motion gate),
slow music tempo.Path H: Echelon State -> Coordinator (iOS -> Mac1, HTTP)
MotionMix EchelonBridge.step()
-> echelon_get_latent(), echelon_get_lexicon()
-> POST :9404/echelon (every 250ms, existing route)
-> multicam-server stores in EchelonState
-> Director loop uses for scoring mode detection
-> Beat quantizer uses for bar-aligned cutsPath I: ML Inference (async, 200ms+ tolerance)
MotionMix or Mac1 triggers inference request
-> HTTP POST to cloud-vm via Tailscale
-> Diffusion model (one-step, CUDA)
-> Response: style transfer params, generated texture
-> Applied to Unity via LUMS control extension
(OR)
-> Mac5 MLX inference (CoreML, local)
-> Response via HTTP to requesting nodeCompound Step 4: Computational Choreography -- Motion to Music
Building on Steps 1-3, detail how body motion becomes music:
The 128D Pipeline (existing, runs on iOS)
RAW INPUTS FEATURE EXTRACTION 128D CANONICAL
----------- ------------------ --------------
CoreMotion accel/gyro --> EchelonBridge Rust FFI --> [0:32] z latent
CoreMotion attitude --> Rust LIM-RPS --> [32:64] velocity
Vision BodyPose --> cachedPoseFeatures --> [63:69] pose
Echelon temporal --> Rust step() --> [69:75] temporal
Modality mask --> Swift bitfield --> [75]
Mocopi 27 bones (LUMM) --> MocopiFeatureExtractor --> [76:100] skeleton
Pocket IMU (local) --> cachedPocketIMU --> [100:102]
Watch (WatchKit) --> cachedWatchFeatures --> [102:104]
LUMF audio ambient (NEW) --> LumeAudioReceiver.swift --> [104:108] (NEW)
LUMS scene state (NEW) --> LumsReceiver.swift --> [108:116] (NEW)
Reserved --> zeros --> [116:128]Motion-to-Music Mapping (existing ParamMapper + SAN)
Layer 1: ParamMapper (reactive, ~30Hz)
rightHeight * 1.5 + velocity boost -> brightness (pads/filter cutoff)
leftHeight * 1.5 + velocity boost -> density (hats/percussion)
bodyEnergy * 2.0 + velocity boost -> energy (overall intensity)
-> filterCutoff = 100 + brightness * 9900 HzLayer 2: Cadence Detection (emergent)
Peak detection on motion signal -> cadencePeakTimes[]
Inter-peak intervals -> motionCadenceHz (0-3.4 Hz)
Interval variance -> rhythmStability (0-1)Layer 3: Motion Accent (impulse)
energyVelocity * 2.1 +
bounceVelocity * 2.4 +
coreVelocity * 1.2 + -> motionAccent (0-1)
legVelocity * 1.1 + Triggers drum fills, transitions
densityVelocity * 0.9 +
brightVelocity * 0.8Layer 4: SAN (Somatic Adaptive Network, 30Hz)
128D canonical -> FAN (Feature-Attention Network)
-> FuseMoE (6 experts, top-2 routing)
-> NHA (Neuro-Harmonic Adapter)
-> TTT (Temporal Texture Transform)
-> FiLM Heads:
- audio_params (BPM target, energy target)
- gesture_confidence
- pattern_intensity
- pattern_variation
-> StrudelWebEngine.sanUpdate({...})
-> Strudel.js pattern modificationLayer 5: ChestFlexDetector (event trigger)
Shoulder width velocity + torso bounce -> flex detection
L/R asymmetry -> flexDirection
Combo system (1x, 2x, 3x COMBO) -> onFlex callback
-> AudioEngine drum triggers: 1x=kick, 2x=kick+clap, 3x=kick+clap+hatMocopi Skeleton -> Music Parameters (specific mapping)
The 24D MocopiFeatureExtractor output at 128D[76:100]:
[76:82] Joint velocities (hips, chest, head, left_hand, right_hand, left_foot)
[82:88] Limb ratios (arm extension L/R, leg extension L/R, spine curl, shoulder width)
[88:94] Temporal derivatives (velocity of velocities, acceleration patterns)
[94:100] Spatial features (center of mass velocity, spread, asymmetry, grounding)These feed into SAN which modulates:
- BPM target: High leg velocity + high cadence = faster tempo
- Energy target: High overall velocity + high spread = higher energy
- Pattern variation: High asymmetry + high acceleration = more complex patterns
- Gesture confidence: Stable cadence + low noise = higher confidence = stronger music response
Compound Step 5: Echelon Multi-Device Coordination
Building on Steps 1-4, detail how Echelon coordinates the mesh:
### Current State
Echelon on iOS computes: latent state (16D), lexicon (tension, divergence), section state (0-6), UI state (orb, spine, horizon). This is posted to :9404/echelon every ~250ms. The multicam server's director loop uses EchelonState for:
- Scoring mode detection (performance vs creator vs photoshoot)
- Beat-quantized cuts (BeatQuantizer aligns cuts to bar boundaries)
- Temporal intelligence (adaptive cut timing based on energy arc)
### What Needs to Happen for LUME
The existing EchelonState on the multicam server needs to flow to K11 Unity so visuals can react to the global performance state, not just local sensor data.
New path: Echelon -> K11 Visual Sync
iOS EchelonBridge.step()
-> POST :9404/echelon (existing)
-> multicam-server EchelonState (existing)
-> NEW: /echelon WS broadcast to subscribers
-> K11 Python echelon_relay.py (NEW, ~60 lines)
subscribes to :9404/director/ws
extracts echelon events
publishes LUME_ECH :9704 (NEW, ~48B, 10Hz)
-> Unity LumeEchelonReceiver.cs (NEW, ~80 lines)
section_state, tension, tempo_bpm, bar_phase
-> _LumeSection, _LumeTension, _LumeTempo globals
-> visual presets auto-switch on section transitions
-> beat-synced particle bursts on bar boundaries### Beat Clock Synchronization
The multicam server already tracks beat phase via BeatQuantizer. Rather than implementing Ableton Link across platforms (complex, untested on Windows), use a simpler approach:
Bar-fire relay: The iOS app posts to :9404/bar-fire on every bar boundary. The server broadcasts this to all /director/ws subscribers. K11's echelon_relay picks this up and publishes a LUME_ECH datagram with updated bar_phase. Unity's LumeEchelonReceiver uses this for beat-quantized visual events.
Effective beat sync accuracy: ~50-100ms (HTTP -> WS -> UDP). Good enough for bar-aligned transitions (bars are 1-4 seconds). NOT good enough for beat-aligned particle hits (those should be driven by local LUMF transient detection instead).
Compound Step 6: The Role of MotionMix LiveDirector
Building on Steps 1-5, clarify the LiveDirector's role:
LiveDirector (multicam server :9404) in LUME Context
The LiveDirector is NOT needed for LUME's visual output (K11 Unity handles that locally). The LiveDirector IS needed for:
1. Multi-camera switching during recording. If iPhones are also capturing the performance, the director loop scores and switches between camera feeds. This runs independently of K11 visuals.
2. Session management. /session/start triggers all devices to enter performance mode. /session/end stops everything. K11 subscribes to session events via echelon_relay.
3. Stills capture. /capture-photo triggers connected iPhones to capture stills. ShootView iPad consumes the stills gallery. Orthogonal to K11 visuals.
4. Performance state hub. All devices report their state. The director provides the single-source-of-truth for "what's happening right now" that any consumer can query.
Port Map (complete)
| Port | Protocol | Format | Producer | Consumer(s) | Plane |
|---|---|---|---|---|---|
| 9700 | UDP | LUMD/LUME | pointcloud_pub.py (K11) | Unity LumeUdpReceiver (K11) | Data |
| 9701 | UDP | LUMF | audio_pub.py (K11) | Unity LumeAudioFftReceiver (K11), MotionMix iOS | Data |
| 9702 | UDP | LUMM | mocopi_bridge.py (K11) | Unity LumeMocopiReceiver (K11), MotionMix iOS | Data |
| 9703 | UDP | LUMS | Unity LumeSystemBus (K11) | MotionMix iOS | Data |
| 9704 | UDP | LUME_ECH | echelon_relay.py (K11) | Unity LumeEchelonReceiver (K11) | Data |
| 9404 | HTTP/WS | JSON | multicam-server (Mac1) | All devices | Control |
| 9600 | HTTP/WS | Strudel | Mac5 Strudel runtime | MotionMix iOS (writer) | Control |
| 8081 | HTTP | MJPEG | LiveStreamServer (iOS) | Web browsers | Control |
| 12351 | UDP | Sony binary | Sony mocopi PC app (K11) | mocopi_bridge.py (K11 localhost) | Data |
Compound Step 7: What Exists vs What Needs to Be Written
Building on Steps 1-6, audit the code inventory:
### EXISTS AND WORKS (no changes needed)
| Component | Location | Lines | Status |
|-----------|----------|-------|--------|
| pointcloud_pub.py | lume-commerce/software/demo/ | 556 | Production |
| audio_pub.py | lume-commerce/software/demo/ | 315 | Production |
| lume_packet_inspector.py | lume-commerce/software/demo/ | ~200 | Production |
| lume_wire_format.md | lume-commerce/software/demo/ | 120 | Production |
| 10 Unity C# components | Assets/Scripts/ | ~2400 | Production |
| 2 compute shaders | Assets/Shaders/ | ~300 | Production |
| 3 editor scripts | Assets/Editor/ | ~300 | Production |
| 5 VFX preset ScriptableObjects | Assets/Scripts/Vfx/ | ~200 | Production |
| 37 Python tests | tests/ | ~800 | Production |
| EchelonBridge.swift | MotionMixApp/Services/ | ~250 | Production |
| ParamMapper.swift | MotionMixApp/Services/ | 218 | Production |
| ChestFlexDetector.swift | MotionMixApp/Services/ | 330 | Production |
| StrudelWebEngine.swift | MotionMixApp/Services/ | 236 | Production |
| AudioEngine.swift | MotionMixApp/Services/ | ~400 | Production |
| LiveStreamServer.swift | MotionMixApp/Services/ | 392 | Production |
| MocopiFeatureExtractor | MotionMixApp/ | ~150 | Production |
| multicam-server main.rs | MotionMix/multicam-server/ | 10678 | Production |
| mocopi-bridge.py (old) | MotionMix/server/ | 256 | JSON output, needs LUMM rewrite |
| libechelon_ios.a + 20 crates | Comp-Core/cc-echelon/ | ~50K | Production (iOS only) |
### NEEDS TO BE WRITTEN (new code)
| Component | Location | Est. Lines | Purpose |
|-----------|----------|-----------|---------|
| mocopi_bridge.py (LUMM) | lume-commerce/software/demo/ | ~200 | Rewrite: Sony :12351 -> LUMM :9702 |
| LumeMocopiReceiver.cs | Unity Assets/Scripts/ | ~120 | Consume LUMM, expose bone data |
| LumeSystemBus.cs | Unity Assets/Scripts/ | ~100 | Publish LUMS :9703 derived state |
| LumeEchelonReceiver.cs | Unity Assets/Scripts/ | ~80 | Consume LUME_ECH :9704 |
| echelon_relay.py | lume-commerce/software/demo/ | ~80 | WS :9404 -> LUME_ECH :9704 relay |
| LummReceiver.swift | MotionMixApp/Services/ | ~80 | Consume LUMM on iOS |
| LumsReceiver.swift | MotionMixApp/Services/ | ~60 | Consume LUMS on iOS |
| LumeAudioReceiver.swift | MotionMixApp/Services/ | ~40 | Consume LUMF on iOS |
| install-services.ps1 | scripts/k11/ | ~100 | NSSM service registration |
| health-check.ps1 | scripts/k11/ | ~80 | Windows diagnostic tool |
| firewall-rules.ps1 | scripts/k11/ | ~20 | UDP port allow rules |
| test_mocopi_bridge.py | tests/ | ~150 | LUMM bridge tests |
| test_echelon_relay.py | tests/ | ~80 | Relay tests |
| lume-k11.json | config/ | ~30 | K11 deployment config |
| LUMM + LUMS + LUME_ECH wire spec | lume_wire_format.md | ~80 | Wire format documentation |
Total new code: ~1,300 lines across 15 files.
Total unchanged existing code: ~67,000+ lines.
### NEEDS MODIFICATION (small changes to existing code)
| Component | Change | Est. Lines |
|-----------|--------|-----------|
| lume_packet_inspector.py | Add LUMM + LUMS + LUME_ECH magic | +10 |
| LumeWaveAutoWire.cs | Wire new receivers in auto-wire menu | +20 |
| LumeCalibrationPanel.cs | Add mocopi + echelon tunables to F12 | +30 |
| EchelonBridge.swift | Wire LummReceiver + LumsReceiver + LumeAudioReceiver | +40 |
| MotionMixApp.swift | Initialize new receivers | +15 |
Compound Step 8: The Complete System Diagram
+============================================================================+
| LUME FULL SYSTEM ARCHITECTURE |
| K11 as Central Hub |
+============================================================================+
DATA PLANE (UDP, real-time, <50ms)
====================================
+--[ K11 Pod (Windows 11) ]------------------------------------------+
| |
| HARDWARE SENSORS |
| +------------------+ +------------------+ +------------------+ |
| | Femto Bolt | | UMA-8 Mic Array | | Sony mocopi | |
| | USB-C, 640x576 | | USB-A, 8ch 48kHz | | 6x BLE IMU | |
| | 30fps ToF depth | | | | via Sony PC app | |
| +--------+---------+ +--------+---------+ +--------+---------+ |
| | | | |
| PYTHON PUBLISHERS (NSSM services, pre-login auto-start) |
| +--------v---------+ +--------v---------+ +--------v---------+ |
| | pointcloud_pub.py| | audio_pub.py | | mocopi_bridge.py | |
| | --raw-depth | | --mic --ch 8 | | :12351 -> LUMM | |
| | LUMD :9700 | | LUMF :9701 | | LUMM :9702 | |
| | 40B hdr, 716px/ | | 84B fixed, 60Hz | | 772B fixed, 50Hz| |
| | chunk, 30fps | | | | | |
| +--------+---------+ +--------+---------+ +--------+---------+ |
| | | | |
| +-------> ALL bind [ip] (Tailscale peers receive) <---+|
| | | | |
| UNITY 6000.3.2f1 (Task Scheduler at-login, Vulkan/DX12) |
| +--------------------------------------------------------------+ |
| | | |
| | INPUT LAYER | |
| | [100] LumeUdpReceiver <---------- :9700 LUMD | |
| | [200] LumeAudioFftReceiver <----- :9701 LUMF | |
| | [100] LumeMocopiReceiver <------- :9702 LUMM [NEW] | |
| | [100] LumeEchelonReceiver <------ :9704 LUME_ECH [NEW] | |
| | | |
| | PROCESSING LAYER | |
| | [0] LumeDepthReprojector (GPU pinhole reproject) | |
| | [0] LumeOpticalFlow (frame-diff + LK dense flow) | |
| | [150] LumeMotionGate (motion -> _LumeTimeScale) | |
| | [100] LumeCalibration (sensor calibration data) | |
| | | |
| | OUTPUT LAYER (VISUAL) | |
| | [0] LumePointRenderer (instanced draw, Cloud/Depth) | |
| | [0] LumeAudioReactor (outline flash, HSV jitter) | |
| | [210] LumeVfxRuntimeBridge (globals -> VFX Graph) | |
| | [220] LumeTransientForcePusher (beat impulse events) | |
| | [0] LumeProceduralTunnel (6144-point helix) | |
| | [--] LumeCalibrationPanel (F12 IMGUI, 16 tunables) | |
| | | |
| | OUTPUT LAYER (NETWORK) [NEW] | |
| | [250] LumeSystemBus -> LUMS :9703 (scene state, 30Hz) | |
| | | |
| | -> URP 17 -> HDMI OUT -> DISPLAY | |
| +--------------------------------------------------------------+ |
| |
| RELAY (bridges control plane to data plane) |
| +--------------------------------------------------------------+ |
| | echelon_relay.py [NEW] | |
| | Subscribes :9404/director/ws (Tailscale) | |
| | Extracts echelon events + bar_fire | |
| | Publishes LUME_ECH :9704 (loopback, 10Hz) | |
| +--------------------------------------------------------------+ |
+----------------------------------------------------------------------+
| | | |
| LUMD :9700 | LUMF :9701 | LUMM :9702 | LUMS :9703
| (depth) | (audio) | (skeleton) | (scene state)
| | | |
v v v v
| | | Tailscale mesh
| +-------+------+ |
| | |
| v v
+--[ MotionMix iOS (iPhone 16 Plus) ]----------------------------+
| |
| SENSOR CONSUMERS |
| LummReceiver.swift <- :9702 LUMM (mocopi skeleton) [NEW] |
| LumeAudioReceiver.swift <- :9701 LUMF (ambient audio) [NEW] |
| LumsReceiver.swift <- :9703 LUMS (scene state) [NEW] |
| |
| LOCAL SENSORS |
| CoreMotion -> accel, gyro, attitude, gravity (60Hz) |
| Vision -> body pose landmarks (30Hz) |
| Camera -> LiveStreamServer :8081 MJPEG |
| |
| INTELLIGENCE PIPELINE |
| +------------------------------------------------------------+ |
| | 128D CANONICAL VECTOR ASSEMBLY | |
| | [0:75] <- EchelonBridge (Rust FFI, CoreMotion+Vision) | |
| | [75] <- modality mask (cam+pocket+mocopi+watch) | |
| | [76:100] <- MocopiFeatureExtractor (from LUMM) | |
| | [100:102]<- cachedPocketIMU (local) | |
| | [102:104]<- cachedWatchFeatures (WatchKit) | |
| | [104:108]<- LumeAudioReceiver (LUMF ambient features) | |
| | [108:116]<- LumsReceiver (K11 scene state) | |
| +------------------------------------------------------------+ |
| | SAN Pipeline (30Hz, 135K params) | |
| | FAN -> FuseMoE -> NHA -> TTT -> FiLM Heads | |
| | -> bpm_target, energy_target, gesture_confidence | |
| | -> pattern_intensity, pattern_variation | |
| +------------------------------------------------------------+ |
| | ParamMapper (30Hz) | |
| | -> brightness (pads), density (hats), energy (overall) | |
| | -> filterCutoff, cadenceHz, rhythmStability, motionAccent | |
| +------------------------------------------------------------+ |
| | ChestFlexDetector (30Hz) | |
| | -> flex events: 1x kick, 2x kick+clap, 3x COMBO | |
| +------------------------------------------------------------+ |
| |
| MUSIC OUTPUT |
| StrudelWebEngine -> Mac5 :9600 (SAN params + patterns) |
| AudioEngine (local fallback if Mac5 unreachable) |
| |
| CONTROL PLANE |
| POST :9404/echelon (250ms, echelon state) |
| POST :9404/bar-fire (on bar boundary) |
| POST :9404/pose/:id (pose joints) |
+------------------------------------------------------------------+
|
| SAN params (10Hz to webview, 2Hz POST to Mac5)
v
+--[ Mac5 (macOS, M-series) ]--------------------------------------+
| Strudel.js :9600 (real @strudel/web + HD1 drum samples) |
| Receives sanUpdate({bpm_target, energy_target, ...}) |
| Modifies live Strudel patterns |
| Audio output -> speakers / line out |
| |
| MLX inference (optional): |
| ConditioningEncoder.mlpackage (104D input, needs 128D retrain) |
| DiffusionService (one-step, style generation) |
+------------------------------------------------------------------+
+--[ Mac1 (macOS) ]------------------------------------------------+
| multicam-server :9404 (Rust/Axum, 10678 lines) |
| 70+ routes, director loop (250ms tick) |
| EchelonState, BeatQuantizer, TemporalIntelligence |
| WebSocket hubs: /director/ws, /device/ws, /td/ws |
| Session management: /session/start, /session/end |
| Stills: /capture, /still-upload, /proof/* |
| |
| Role in LUME: coordination hub, NOT data processor |
| Receives: echelon state from iOS, device registrations |
| Broadcasts: director cuts, session events, bar-fire relays |
+------------------------------------------------------------------+
+--[ cloud-vm (Linux, CUDA GPU) ]----------------------------------+
| Diffusion inference API (REST, async) |
| One-step models for visual style transfer |
| Called by Mac5 or Mac1, result applied to Unity via config |
| Latency: 200ms+ (async, not real-time path) |
+------------------------------------------------------------------+
CONTROL PLANE (HTTP/WS, coordination)
======================================
iOS ---POST /echelon, /bar-fire----> Mac1 :9404
iOS ---POST /san 2Hz--------------> Mac5 :9600
Mac1 ---WS /director/ws broadcast--> K11 echelon_relay.py
Mac1 ---WS /director/ws broadcast--> iOS (if subscribed)
Mac1 ---HTTP /session/*------------> All devices
K11 ---POST /register-------------> Mac1 :9404 (optional)Promotion Decision
Promote into a technical note or architecture paper with implementation anchors.
Source Anchor
evo-cube-output/lume-full-system-architecture/stage2-compound.md
Detected Structure
Method · Evaluation · Code Anchors · Architecture · is Stage Research