Computational Choreography — System Overview

Computational Choreography — System Overview

Architecture

The system flows through five main stages:

Sensors → RPS Encoder → GRU Mapper → Bridges → Strudel Sound Engine

1. Sensors

Multi-modal input streams sampled at 100 Hz:
- Mocopi suit — 24-point skeleton tracking
- Polar H10 — heart rate + HRV
- Video pose — MediaPipe/OpenPose fallback
- Mobile accel — phone/tablet backup

Features extracted per frame:
- Motion energy (kinetic sum)
- HR derivative (arousal)
- Spatial symmetry (left/right balance)
- Center of mass trajectory
- Velocity peaks
- Limb angles

2. RPS Encoder (Relational Proximal Synchrony)

Converts multi-channel features into a coherent latent representation `x⁺`:
- Handles missing modalities through hallucination
- Maintains temporal coherence via proximal update
- Outputs normalized latent vector (dim=16-32)

3. GRU Mapper (Reflex Network)

Sliding window (2-4 seconds) of latent vectors → control signals:
- `tempo_scale` — BPM multiplier (0.75–1.5)
- `density` — rhythmic complexity (0–1)
- `tension` — filter/harmonic tension (0–1)
- `brightness` — spectral brightness (0–1)
- `fx_send` — reverb/delay mix (0–1)
- `swing` — rhythmic swing (0–0.6)
- `accent` — dynamic range (0–1)

4. Bridges

Control signals → sound engine protocols:
- MIDI Bridge — CC messages via Web MIDI (IAC Driver)
- OSC Bridge — UDP to SuperCollider/Max
- Link Bridge — Ableton Link sync for phase/tempo

5. Strudel Sound Engine

TidalCycles-inspired live coding environment:
- Web-based, runs in browser
- Receives MIDI CC → modulates pattern parameters
- Four layers: drums, bass, pads, FX
- Real-time pattern evaluation

---

Data Flow

Training Mode

synthetic_sessions → build_dataset → train_rps → train_mapper → checkpoints

Inference Mode

sensor_stream → realtime_loop → bridges → strudel_embed → audio_output

---

Key Concepts

Bar Context

Aggregates per-bar summaries for downstream reasoning:
- Average energy
- Density trend
- Phase stability
- Gesture events

Safety Guards

Clamps, smoothing, and fallback:
- NaN policy: hold last valid value
- Rate limiting: max 50 Hz updates
- Range clamping per `control_map.yaml`

Easing Functions

Converts step changes to musical curves:
- Linear — direct mapping
- Exponential — perceptual scaling (loudness, frequency)
- S-curve — smooth acceleration/deceleration

---

Performance Considerations

• Latency target: < 50 ms sensor-to-sound
• Jitter tolerance: ± 5 ms phase variance
• CPU budget: < 30
• Memory: < 200 MB resident

---

Future Extensions

• Lookahead planner — predicts latent 400 ms ahead for bar-accurate scheduling
• Gesture macros — trigger fills, drops, builds
• Multi-dancer sync — ensemble coordination
• Adaptive training — online learning from sessions