MotionMix — Technical Architecture

> Full system architecture: hardware sensors → Rust engine → neural synthesis → multi-machine rendering > Last updated: 2026-04-16 MotionMix is a real-time motion-to-music synthesis platform. A performer's body movement is captured via wearable sensors and iPhone cameras, processed through a Rust engine (Echelon) into a 128-dimensional canonical vector, fed through a 5-layer neural network (SAN), and output as audio parameters, camera decisions, visual effects, and 3D body rendering across a fleet of devices. ### Sony Mocopi - 27 inertial measurement units worn on body joints - Streams skeleton data at 30Hz via UDP/OSC - Each bone: world-space position [x,y,z] + orientation quaternion [qw,qx,qy,qz] - OSC formats: `/mcp/sklt/joint/N` (per-bone), `/mocopi/skel` (flat 189 floats), `/mcp/BonePos/{name}` (position-only) ### iPhone CoreMotion - Accelerometer, gyroscope, gravity vector, attitude quaternion - 60Hz sampling via CMMotionManager - Feeds the LIM-RPS latent space dynamics in Rust ### iPhone Camera + Vision - AVCaptureSession rear camera - Apple Vision VNDetectHumanBodyPoseRequest extracts 14 body joints - Each joint: normalized [x, y, confidence] - Derived metrics: body energy, bouncing (hip Y oscillation), core motion, leg motion