Implementation Roadmap — Computational Choreography Models

Implementation Roadmap — Computational Choreography Models

This roadmap breaks down the model implementation into actionable tasks with clear acceptance criteria.

---

Phase 0: Foundation & Contracts (Week 1)

### Task 0.1: Data Schema Validation
- [ ] Verify all schema files in `data/schemas/` match specification
- [ ] Create Python validators for each schema type
- [ ] Add unit tests for schema validation
- Files: `sdk/python/validators.py`, `tests/test_schemas.py`
- Acceptance: Can load and validate example parquet files

### Task 0.2: Update Packet Types
- [ ] Enhance `sdk/python/packets.py` with complete types per spec
- Add SensorPacket with all fields
- Add LatentPacket with modality arrays
- Add ControlPacket with all 5 controls
- [ ] Add serialization methods (to_dict, from_dict)
- [ ] Create matching TypeScript types in `sdk/web/`
- Acceptance: All packet types match specification exactly

### Task 0.3: Configuration System
- [ ] Update `configs/control_map.yaml` with complete specification
- Add all 5 controls with min/max/ease/cc
- Add throttle_ms and clamp rules
- [ ] Create config loader in `sdk/python/config.py`
- [ ] Add validation for config files
- Acceptance: Can load configs; invalid configs raise clear errors

---

Phase 1: M0 — Synthetic Data Generation (Week 1-2)

### Task 1.1: Enhanced Synthetic Generator
- [ ] Update `scripts/generate_synthetic_sessions.py` with full spec:
- Add all 11 frame features (motion_energy, motion_freq, jerk, hip_*, symmetry, hr, hr_slope)
- Implement beat_phase and beat_idx computation
- Add realistic HR lag (~3 seconds behind motion_energy)
- Add jerk spikes near phase 0.0 and 0.5
- [ ] Add CLI arguments: --sessions, --minutes, --bpm, --out, --seed
- [ ] Generate meta.json with session metadata
- Acceptance:
- Generates 5 sessions × 3 minutes each
- All columns present and within bounds
- HR visibly lags motion_energy when plotted
- Jerk spikes align with downbeats

### Task 1.2: Data Inspection Tools
- [ ] Create `scripts/inspect_session.py` for quick visualization
- Plot motion_energy, hr, beat_phase overlay
- Show feature histograms
- Check for NaNs and outliers
- Acceptance: Can quickly verify synthetic data quality

### Task 1.3: Teacher Target Generator
- [ ] Create `training/dataloaders/teacher_targets.py`
- Implement rules: tempo_scale ← motion_freq
- density ← motion_energy
- tension ← hr_slope
- brightness ← symmetry
- fx_send ← jerk
- [ ] Add unit tests for each rule
- Acceptance: Teacher targets are reasonable and smooth

---

Phase 2: M1 — RPS Coherence (Week 2-3)

### Task 2.1: Encoder Implementation
- [ ] Update `models/rps/encoders.py` with spec:
- `TinyEncoder(in_dim, out_dim, hidden=128)` MLP with ReLU
- `make_encoders()` returning ModuleDict with 4 encoders
- mot: 6→64
- hr: 2→16
- aud: 2→16
- ctx: 1→8
- [ ] Add forward pass tests
- Acceptance: Forward pass returns correct shapes; no NaNs

### Task 2.2: Translator Implementation
- [ ] Update `models/rps/translators.py` with spec:
- Linear translators: 104→each modality size
- `make_translators()` returning ModuleDict
- Spectral norm utility function
- [ ] Add spectral norm clamping (≤0.9) utility
- Acceptance: Translators produce finite outputs; spectral norm ≤0.9

### Task 2.3: Proximal Step Implementation
- [ ] Update `models/rps/prox_update.py` with spec:
- `rps_prox_step(latents: dict, translators, alpha=0.2)`
- Concatenate all modalities → predict each → blend
- [ ] Add unit test: prox reduces cross-prediction error
- Acceptance: Prox step is stable; reduces cross-pred MSE

### Task 2.4: Modality Dropout
- [ ] Add `modality_dropout(latents, p=0.2)` function
- [ ] Test that training with dropout enables hallucination
- Acceptance: Can mask motion; prox reconstructs with reasonable magnitude

### Task 2.5: RPS Trainer
- [ ] Create complete `training/trainers/train_rps.py`:
- Load frames from data/raw/
- Extract modality inputs (motion, hr, aud, ctx)
- Forward encoders with modality dropout
- Run prox step
- Compute cross-prediction MSE loss
- Optimizer: Adam lr=1e-3, batch=256, epochs=5
- Spectral norm clamp after each step
- Save checkpoint to `training/checkpoints/rps/rps_light.pt`
- [ ] Add training visualization (loss curves)
- [ ] Add checkpoint save/load utilities
- Acceptance:
- Loss decreases steadily
- No NaNs or explosions
- Checkpoint saves successfully
- Missing-modality test passes

### Task 2.6: Generate Latent Timelines
- [ ] Add option to dump latent timelines to parquet
- [ ] Create `data/processed/<session>_latent.parquet` files
- Acceptance: Latent files match schema; can be loaded for M3 training

---

Phase 3: M2 — Latent Normalizer (Week 3)

### Task 3.1: Normalizer Implementation
- [ ] Create `models/rps/normalizer.py`:
- `class LatentNormalizer` with running mean/var per dim
- `update(latents)` method
- `apply(latents)` method (z-score normalization)
- `state_dict()` and `load_state_dict()` for persistence
- [ ] Add unit tests: normalize → denormalize recovers original
- Acceptance: Normalizer is stable; saved with RPS checkpoint

### Task 3.2: Integration with Training
- [ ] Add normalizer to `train_rps.py`
- [ ] Compute and save normalization stats with checkpoint
- Acceptance: Normalized latents improve M3 training stability

---

Phase 4: M3 — GRU Reflex Mapper (Week 4-5)

### Task 4.1: GRU Mapper Model
- [ ] Update `models/mapper/gru_mapper.py`:
- `class GRUMapper(x_dim=104, hidden=96, out_dim=5)`
- Input: (B, T, 104) → Output: (B, 5)
- Output order: [tempo_scale, density, tension, brightness, fx_send]
- [ ] Add forward pass unit tests
- [ ] Profile inference time (target < 2ms CPU)
- Acceptance: Forward pass is fast and stable

### Task 4.2: Loss Functions
- [ ] Update `models/mapper/losses.py`:
- `smoothness_loss(ctrl_t, ctrl_prev)` — L1 on deltas
- `beat_regularizer(beat_phase_seq, delta_ctrl_seq)` — penalize off-beat changes
- [ ] Add unit tests for each loss
- Acceptance: Losses are reasonable and differentiable

### Task 4.3: Windowing Utilities
- [ ] Update `training/dataloaders/sessions.py`:
- `SessionWindowDataset` for sliding windows
- Window size: 20-50 frames (0.5-1.0s)
- Efficient batching
- [ ] Add tests for window extraction
- Acceptance: Can efficiently load windows from multiple sessions

### Task 4.4: Mapper Trainer
- [ ] Create complete `training/trainers/train_mapper.py`:
- Load frames and compute latents (frozen M1 encoders)
- Create sliding windows of latents
- Compute teacher targets from frames
- Train GRU with combined loss:
- MSE(pred, teacher)
- + λ_smooth × smoothness
- + λ_beat × beat_regularizer
- Optimizer: Adam lr=1e-3, batch=64, epochs=5
- Gradient clip 1.0
- Save checkpoint to `training/checkpoints/mapper/gru_mapper.pt`
- [ ] Add training visualization
- Acceptance:
- Training loss decreases
- Smoothness is reasonable
- Controls don't clip excessively
- Inference time < 2ms

### Task 4.5: Generate Control Timelines
- [ ] Add option to save control predictions to parquet
- [ ] Create `data/processed/<session>_controls.parquet` files
- Acceptance: Control files match schema; ready for evaluation

---

Phase 5: Realtime Loop & Inference (Week 5-6)

### Task 5.1: Core Realtime Loop
- [ ] Update `inference/realtime_loop.py` with complete pipeline:
1. Read frames (from file or stream)
2. Forward through M1 encoders → prox → latents
3. Apply normalization (M2)
4. Gather window → M3 → raw controls
5. Apply safety guards and throttling
6. Output to bridge
- [ ] Add session recording capability
- [ ] Add profiling/latency monitoring
- Acceptance:
- Runs stable > 5 minutes
- Average latency < 5ms for M1+M3
- No memory leaks

### Task 5.2: Controller Components
- [ ] Update `inference/controller/throttling.py`:
- Fixed cadence output (20-50ms)
- Skip frames if necessary
- [ ] Update `inference/controller/easing.py`:
- Exponential, s-curve, linear easing
- Apply per-control based on config
- [ ] Update `inference/controller/transport.py`:
- BPM/phase management
- Tempo leader policy (you vs. Strudel)
- Acceptance: Controls are smooth and musical

### Task 5.3: Safety Guards
- [ ] Update `inference/safety_guards.py`:
- Clamp controls to configured ranges
- Rate limiting (max change per second)
- NaN handling (hold last valid)
- Smoothing window
- [ ] Add unit tests for edge cases
- Acceptance: System is stable even with corrupted inputs

---

Phase 6: Bridges & Strudel Integration (Week 6-7)

### Task 6.1: MIDI Bridge
- [ ] Update `inference/bridges/midi_out.py`:
- Load CC mapping from `configs/control_map.yaml`
- Open specified MIDI device
- Send CC messages for each control
- Apply latency offset
- Device selection and error handling
- [ ] Add mock MIDI device for testing
- Acceptance: Can send CC to IAC Driver; monitor shows messages

### Task 6.2: OSC Bridge (Alternative)
- [ ] Update `inference/bridges/osc_out.py`:
- Send controls as OSC messages
- Configurable host/port
- Acceptance: Can send to SuperCollider/Max

### Task 6.3: Strudel Patches
- [ ] Update `sound/strudel/patches/core_patch.strudel.md`:
- Complete working patch using midiin()
- Map all 5 CC numbers to pattern parameters
- Test with live CC input
- [ ] Update individual pattern files (drums, bass, pads, fx)
- Acceptance: Strudel responds audibly to CC changes

### Task 6.4: Latency Calibration
- [ ] Create `scripts/measure_latency.py`:
- Send test CC pulse
- Record audio
- Measure round-trip time
- Generate calibration config
- [ ] Document calibration procedure
- Acceptance: Can measure and configure latency offset

---

Phase 7: Evaluation & Metrics (Week 7-8)

### Task 7.1: Phase Coherence Metric
- [ ] Update `evaluation/metrics/phase_coherence.py`:
- Extract control peaks (density)
- Measure distance to beat_phase 0
- Report score 0-1
- [ ] Add unit tests with synthetic data
- Acceptance: Metric is stable; target > 0.6 on synthetic

### Task 7.2: Energy Correlation Metric
- [ ] Update `evaluation/metrics/energy_correlation.py`:
- Compute Pearson r between motion_energy and density
- Handle lag search (±100ms)
- Acceptance: Target > 0.3 on synthetic

### Task 7.3: Drift Stability Metric
- [ ] Update `evaluation/metrics/drift_stability.py`:
- Measure phase drift over time
- Report stability score
- Acceptance: Target ~ 0.95-1.0

### Task 7.4: Prediction Error (for M4)
- [ ] Update `evaluation/metrics/prediction_error.py`:
- MAE between predicted and actual latent
- Compare to baseline
- Acceptance: Lower than naive repeat-last

### Task 7.5: Groove Report
- [ ] Update `evaluation/groove_report.py`:
- Load session with frames + controls
- Compute all metrics
- Generate JSON/markdown report
- Include plots
- [ ] Add CLI interface
- Acceptance: Can run on any session; produces clear report

### Task 7.6: Plotting Tools
- [ ] Update `evaluation/plots/phase_histogram.py`
- [ ] Update `evaluation/plots/energy_timeseries.py`
- [ ] Add control traces visualization
- Acceptance: Plots are publication-ready

---

Phase 8: Integration & Testing (Week 8-9)

### Task 8.1: End-to-End Test
- [ ] Create `tests/test_end_to_end.py`:
- Generate synthetic session
- Train M1
- Train M3
- Run inference loop
- Generate metrics report
- All automated
- Acceptance: Full pipeline runs without errors

### Task 8.2: Makefile Integration
- [ ] Update Makefile with all commands:
- `make generate-data` → M0
- `make train-rps` → M1
- `make train-mapper` → M3
- `make run-sim` → realtime loop (file mode)
- `make run-live` → realtime loop (sensor mode)
- `make evaluate` → metrics report
- Acceptance: Can run entire workflow with make commands

### Task 8.3: Documentation
- [ ] Create `docs/training_guide.md`:
- How to generate data
- How to train models
- How to interpret metrics
- Hyperparameter tuning guide
- [ ] Create `docs/inference_guide.md`:
- How to run realtime loop
- How to configure MIDI/OSC
- Latency optimization tips
- Acceptance: Documentation is clear and complete

### Task 8.4: Requirements & Dependencies
- [ ] Update `requirements.txt` with exact versions:
- torch>=2.0.0
- pyarrow (parquet)
- pandas
- scipy (for metrics)
- matplotlib (for plots)
- mido or python-rtmidi (MIDI)
- python-osc
- [ ] Test installation on clean environment
- Acceptance: `pip install -r requirements.txt` works

---

Phase 9: Optional Models (Week 10+)

### Task 9.1: M4 — Look-Ahead Predictor
- [ ] Implement `models/lookahead/temporal_cnn.py`
- [ ] Create `training/trainers/train_lookahead.py`
- [ ] Integrate with realtime loop for bar scheduling
- [ ] Evaluate prediction MAE
- Acceptance: MAE < baseline; improves bar-accurate events

### Task 9.2: M5 — Gesture Detector
- [ ] Implement `models/gesture/cnn_detector.py`
- [ ] Add gesture labels to synthetic generator
- [ ] Create `training/trainers/train_gesture.py`
- [ ] Add gesture triggers to control flow
- Acceptance: F1 ≥ 0.85 per class

### Task 9.3: M6 — Reward Model
- [ ] Implement `models/reward/mlp_reward.py`
- [ ] Create preference collection interface
- [ ] Train with pairwise ranking loss
- Acceptance: AUC ≥ 0.75 on held-out pairs

### Task 9.4: M7 — Bar Planner
- [ ] Implement `inference/planner/rule_policy.py`
- [ ] Add bar context aggregation
- [ ] Test macro triggers on bar boundaries
- Acceptance: Events land on correct bars; no spam

---

Phase 10: Polish & Deployment (Week 11-12)

### Task 10.1: Performance Optimization
- [ ] Profile critical paths
- [ ] Optimize M1+M3 inference (JIT, quantization)
- [ ] Reduce memory footprint
- Acceptance: Latency < 50ms end-to-end on target hardware

### Task 10.2: Error Handling
- [ ] Add comprehensive error messages
- [ ] Graceful degradation for missing modalities
- [ ] Recovery procedures in realtime loop
- Acceptance: System handles edge cases gracefully

### Task 10.3: Logging & Monitoring
- [ ] Add structured logging throughout
- [ ] Add performance metrics logging
- [ ] Create monitoring dashboard (optional)
- Acceptance: Can debug issues from logs alone

### Task 10.4: Version Tagging
- [ ] Tag v0.1.0 with working M1+M3
- [ ] Create release notes
- [ ] Archive baseline checkpoints
- Acceptance: Clean release ready for iteration

---

Success Criteria Summary

M0: 5 synthetic sessions with realistic features
M1: Cross-pred loss < 0.1; missing-modality test passes
M2: Normalizer reduces M3 loss variance
M3: Teaching loss ↓; smoothness reasonable; inference < 2ms
Inference: Stable > 5 min; latency < 50ms end-to-end
Bridges: Strudel responds audibly to motion
Metrics: Phase coherence > 0.6; energy corr > 0.3; drift ~ 1.0

---

Estimated Timeline

• Weeks 1-2: Foundation + M0 (synthetic data)
• Weeks 3-4: M1 (RPS) + M2 (normalizer)
• Weeks 4-5: M3 (mapper)
• Weeks 5-7: Inference + bridges + Strudel
• Weeks 7-8: Evaluation + integration
• Weeks 8-9: Testing + documentation
• Weeks 10+: Optional models (M4-M7)
• Weeks 11-12: Polish + deployment

Total: ~12 weeks to production-ready v0.1 with core models
Optional: +4-6 weeks for advanced features (M4-M7)