Path D: Unity-as-Middleware -- Unity on K11 Becomes the Central Bus

Path D: Unity-as-Middleware -- Unity on K11 Becomes the Central Bus

Concept

Unity is not just a renderer. It becomes the system's central message bus. All sensor data flows into Unity. Unity extracts features (motion, audio, depth), makes all reactive decisions locally, and ALSO publishes derived features outward to other consumers via UDP. Unity becomes both consumer AND producer. A new "LumeSystemBus" C# component aggregates all signals and publishes a unified state packet at 30Hz that any mesh node can subscribe to.

Architecture

K11 (Unity = Central Bus)
+-----------------------------------------------------------+
| SENSORS -> PUBLISHERS -> Unity (all local UDP)             |
|                                                             |
| Unity Components:                                          |
| [Input Layer]                                              |
|   LumeUdpReceiver    <- LUMD :9700 (depth)                |
|   LumeAudioFftReceiver <- LUMF :9701 (audio)              |
|   LumeMocopiReceiver <- LUMM :9702 (skeleton) [NEW]       |
|                                                             |
| [Processing Layer]                                         |
|   LumeDepthReprojector -> world-space point cloud          |
|   LumeOpticalFlow -> motion scalar + flow field            |
|   LumeMotionGate -> time scale from motion                 |
|   LumeMocopiFeatures -> bone velocities/ratios [NEW]      |
|   LumeChoreographyMapper -> motion-to-visual params [NEW] |
|                                                             |
| [Output Layer - Visual]                                    |
|   LumePointRenderer -> HDMI display                        |
|   LumeVfxRuntimeBridge -> VFX Graph particles              |
|   LumeTransientForcePusher -> impulse events               |
|   LumeAudioReactor -> outline flash/jitter                 |
|                                                             |
| [Output Layer - Network] [NEW]                             |
|   LumeSystemBus -> publishes LUMS :9703 (state packet)    |
|     Contains: motion_energy, skeleton_features[24],        |
|     audio_rms, band_levels[4], transient, flow_magnitude,  |
|     time_scale, active_preset_id                           |
|     84-byte fixed datagram at 30Hz to [ip]             |
|                                                             |
+-----------------------------------------------------------+
        |
        | LUMS :9703 (derived state)
        v
MotionMix iOS              Mac5 Strudel
+--------------------+    +------------------+
| Consumes LUMS      |    | Consumes LUMS    |
| Maps to 128D [104:]|    | Maps to params   |
| + local sensors    |    | for Strudel.js   |
| Drives SAN + audio |    | Audio synthesis   |
+--------------------+    +------------------+

Strengths

• Zero latency for visual path. All sensor -> visual processing is local loopback.
• Unity already does feature extraction. LumeOpticalFlow computes motion. LumeAudioFftReceiver computes audio features. LumeMotionGate computes time scale. Unity is ALREADY the bus; this path just makes it explicit.
• New LUMS wire format is cheap. 84 bytes at 30Hz is trivial network load. Derived features are more useful than raw sensor data for remote consumers.
• Simplifies MotionMix's job. Instead of consuming 3 raw streams (LUMD, LUMF, LUMM), MotionMix consumes 1 derived stream (LUMS). Feature extraction happens once on K11, not redundantly on every consumer.
• Consistent with LUME wire format family. LUMS (0x4C554D53) extends the existing protocol.

Weaknesses

• Unity main thread bottleneck. All C# components run on Unity's main thread (Update/LateUpdate). Adding more processing risks frame drops below 60fps on Radeon 780M.
• C# is the wrong language for motion intelligence. ParamMapper's velocity detection, cadence detection, and rhythm stability algorithms are elegant in Swift. Porting to C# loses the tight integration with Vision/CoreMotion and the SAN pipeline.
• Mocopi feature extraction in Unity is weak. MocopiFeatureExtractor in Swift uses specific joint velocity/limb ratio algorithms tuned for the 128D canonical vector. A C# version would be a parallel implementation with no shared test suite.
• Unity is not a message broker. It doesn't handle subscriptions, backpressure, or reliable delivery. Bolting pub-sub onto Unity's update loop is architecturally wrong.
• Loses the 128D canonical vector. The canonical vector is computed by Echelon (Rust via iOS FFI). Unity has no access to the latent state, SAN outputs, lexicon, or section state.

Verdict

PARTIAL ADOPT. The insight that Unity should PUBLISH derived features (not just consume raw ones) is correct. LUMS as a derived state packet is a good idea. But Unity should not try to be a general-purpose intelligence layer or replace MotionMix's motion-to-music mapping. Unity's job: consume sensor data, render visuals, publish derived visual-scene state.