๐ฑ EchelonCapture Restructuring Plan
**Date**: December 21, 2025 **Status**: ๐ Planning Phase **Context**: Multi-sensor ecosystem with Mocopi + Strudel music generation
Full Public Reader
๐ฑ EchelonCapture Restructuring Plan
Date: December 21, 2025
Status: ๐ Planning Phase
Context: Multi-sensor ecosystem with Mocopi + Strudel music generation
---
๐ Executive Summary
With the addition of Sony Mocopi sensors (6 body-mounted IMUs) and the existing Strudel music generation engine, we need to restructure EchelonCapture to fit into a coherent multi-sensor, multi-app ecosystem.
### Current Situation
- 3 iOS Apps: EchelonCapture, cc-handguard, TrajectoryOS
- Sensors: iPhone (2x), Apple Watch, AirPods, Mocopi (6x new)
- Music Generation: Strudel (JavaScript/Web Audio) + optional Neural Audio Server
- Backend: cc-mcs Cloud Run (Rust) for latent physics + Computational Rehearsal
### Key Question from User
> "what's also to be considered from the iOS perspectives how many iPhone applications are to we to be creating as you know we have Strudel and the music generation just wondering if that should be all together"
### Answer
Strudel music generation runs in JavaScript (browser or WKWebView). We have 3 distinct iOS apps with different purposes:
1. EchelonCapture - Motion capture + local Strudel playback
2. cc-handguard - Behavioral intervention (nail-biting detection)
3. TrajectoryOS - Life trajectory optimization
Recommendation: Keep them separate. Each has a distinct use case and user journey.
---
๐ฏ What is Strudel?
From [cc-rehearsal.md](apps/ios/EchelonCapture/cc-rehearsal.md):
> Strudel is a JavaScript music runtime - a pattern-based music engine inspired by live coding.
>
> - Declarative: music is code (patterns, transforms, functions)
> - Hot-swappable: you can change the pattern graph while sound is playing
> - Time-aware: thinks in cycles, measures, "every 4", "sometimes", etc.
> - Browser-native: runs using Web Audio API (Tone.js)
Where Strudel Runs on iOS
Three integration patterns:
1. Browser-based (Current cc-dashboard): Strudel runs in Safari, audio plays from browser
2. WKWebView embedded (Recommended for EchelonCapture): Native Swift app with embedded web view running Strudel
3. Native Swift reimplementation: Recreate pattern engine in AVAudioEngine (hard, not recommended)
Recommended for EchelonCapture: Option 2 (WKWebView)
- Native Swift for sensors (CoreMotion, WatchConnectivity)
- Embedded WKWebView for Strudel JavaScript runtime
- Local audio playback via Web Audio
- App Store distribution
- Best of both worlds
---
๐๏ธ Updated Architecture with Strudel
Sensor โ Latent โ Music Flow
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ FULL SYSTEM ARCHITECTURE โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ โ
โ ๐ฑ EchelonCapture (iOS) ๐ญ Mocopi Bridge (Python) โ
โ โโ iPhone Left (Swift) โโ hip โ
โ โโ iPhone Right โโ head โ
โ โโ Apple Watch โโ left_hand โ
โ โโ AirPods โโ right_hand โ
โ โโ ๐ต Strudel (WKWebView) โโ left_foot โ
โ โโ right_foot โ
โ โ โ โ
โ POST /api/data POST /api/data โ
โ โ โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ cc-mcs Cloud Run Backend (Rust) โ โ
โ โ - Sensor fusion (all 10+ devices) โ โ
โ โ - LIM-RPS latent physics (25D state) โ โ
โ โ - Computational Rehearsal (predict future) โ โ
โ โ - Conductor (musical state machine) โ โ
โ โ - Pattern-coder (trajectory โ edits) โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ โ
โ WebSocket stream โ
โ โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ Client (Browser or WKWebView in EchelonCapture) โ โ
โ โ - Receives latent state (50 Hz) โ โ
โ โ - Receives pattern edits (per bar) โ โ
โ โ - Strudel engine (always-on music runtime) โ โ
โ โ - Web Audio synthesis (speakers/headphones) โ โ
โ โ - Optional: Neural Audio Server phrases (GPU) โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ โ
โ ๐ MUSIC OUTPUT โ
โ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโComputational Rehearsal Loop
Your Body (Moving)
โ
Sensors (iPhone, Mocopi, Watch, AirPods)
โ
cc-mcs Backend (LIM-RPS)
โ
Latent State z(t) [25D: tension, excitement, convergence, ...]
โ
Computational Rehearsal โ Predicted Future z(t+ฮ)
โ
Conductor + Pattern-Coder
โ
Pattern Edits (symbolic commands)
โ
Strudel Runtime (client-side)
โ
๐ต Music (adapts to your predicted movement)
โ
You hear music โ adjust your movement
โ
(loop continues)Key Insight: The system doesn't just react to your current movement - it rehearses your near future and generates music that anticipates where you're going.
---
๐ฏ EchelonCapture's Role with Strudel
### Before Restructuring
EchelonCapture was:
- Data collection tool
- Real-time visualization (Latent Orb, Trajectory)
- Streaming to cc-mcs
- No music generation (music was in cc-dashboard)
### After Restructuring
EchelonCapture becomes:
- Primary performance app for dancers
- Sensor streaming (iPhone + Watch + AirPods)
- Local Strudel music playback (embedded WKWebView)
- Optional recording for training datasets
- No heavy visualizations (simplified UI)
- Mocopi-aware (shows all 10+ sensors)
Why embed Strudel?
- Dancers want to hear music on-device while performing
- Browser-only approach requires separate laptop/tablet for audio
- WKWebView gives native app experience + JavaScript music engine
- Can still use cc-dashboard for DJ/producer view (full visualization)
---
๐ฑ Redesigned EchelonCapture with Strudel
### Core Purpose
Dancer-focused performance app that streams sensors and plays AI-generated music locally.
### What to REMOVE โ
1. Visualization components (no longer needed):
- `PerformanceView.swift` - Latent Orb, Trajectory Overlay
- `VisualizationService.swift` - Real-time graphics
- WebSocket `/visualization` connection
- Embodied Features Panel
- Stats overlay
2. WebSocket for visualization (not needed):
- Remove WS client code
- Keep only HTTP POST for sensor data
### What to KEEP โ
1. Core sensor streaming:
- `MotionStreamer.swift` - CoreMotion at 50-100Hz
- `NetworkService.swift` - HTTP POST to cc-mcs
- Device role selection (LEFT, RIGHT, WATCH, HEAD)
2. Session management (optional):
- `SessionManager.swift` - Session lifecycle
- `StorageService.swift` - Local recording
- `AudioRecorder.swift` - Audio capture
- Session markers
3. Background execution:
- `BackgroundManager.swift` - Keep streaming while screen off
- Background audio mode (for Strudel playback)
### What to ADD โจ
1. Strudel Integration (NEW!):
- Add WKWebView to host Strudel JavaScript engine
- Create `StrudelBridge.swift` - Swift โ JS communication
- WebSocket connection to cc-mcs for latent state + pattern edits
- Local audio playback via Web Audio API
2. Simplified Performance UI:
- Clean, minimal interface
- Large "Start Performance" button
- Connection status (sensors + backend + music)
- Audio level meter (visual feedback that music is playing)
3. Multi-sensor awareness:
- Status showing all active sensors (iPhone + Mocopi)
- Total sensor count display
- Backend health indicator
---
๐จ Proposed New UI Structure
Tab Structure (Simplified)
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ EchelonCapture โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ Perform Record Settings โโ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โ โ PERFORM TAB (NEW!) โ โ
โ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฃ โ
โ โ โ โ
โ โ ๐ต MUSIC: Playing โ โ
โ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
โ โ โ โโโโโโโโโโโโโโโโโโโ โ โ โ
โ โ โ Audio Level Meter โ โ โ
โ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
โ โ โ โ
โ โ ๐ฑ Sensors: 10 active โ โ
โ โ โโ iPhone (This): โ
โ โ
โ โ โโ iPhone Right: โ
โ โ
โ โ โโ Apple Watch: โ
โ โ
โ โ โโ AirPods: โ
โ โ
โ โ โโ Mocopi (6): โ
โ โ
โ โ โ โ
โ โ ๐ Backend: Connected โ โ
โ โ ๐ญ Latent: Streaming โ โ
โ โ โ โ
โ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
โ โ โ โ STOP PERFORMANCE โ โ โ
โ โ โโโโโโโโโโโโโโโโโโโโโโโโโโโ โ โ
โ โ โ โ
โ โ Stats: 12,450 frames sent โ โ
โ โ 98 Hz | 85ms latency โ โ
โ โ โ โ
โ โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ### Perform Tab (NEW - Primary View)
Purpose: Start performance, stream sensors, hear music
Features:
- Large status indicators:
- Music status (Playing/Stopped with audio level meter)
- Sensor count + individual sensor status
- Backend connection health
- Latent stream status
- Single action button:
- "Start Performance" โ starts sensors + Strudel
- "Stop Performance" โ stops everything
- Audio feedback:
- Audio level meter (visual confirmation music is playing)
- Optional: Current section (Groove/Build/Climax/Breakdown)
- Optional: BPM display
- Multi-sensor ecosystem view:
- Shows all active devices (iPhone, Watch, AirPods, Mocopi)
- Clear visual: green checkmarks for active sensors
- Minimal stats (collapsible):
- Frames sent, rate, latency
What's REMOVED from old PerformanceView:
- Latent Orb visualization
- Trajectory overlay
- Embodied features panel
- Complex real-time graphics
### Record Tab (Keep, Simplified)
Purpose: Local session recording for training data
Features (simplified from current):
- Session name
- Start/Stop recording
- Moment markers (good/off/peak/drop)
- Session save
Remove: Heavy UI, complex session browser (move to Settings or separate tab)
### Settings Tab (Enhanced)
Purpose: Configuration + ecosystem status
Features:
- Backend Configuration:
- Backend URL (Cloud Run or localhost)
- Connection test button
- Device Configuration:
- Device role (LEFT, RIGHT, WATCH, HEAD)
- Sensor rate (50/100 Hz)
- Strudel Configuration (NEW):
- Audio volume
- Enable/disable neural audio phrases (optional)
- Music style presets (House, Techno, Ambient, etc.)
- Sensor Ecosystem (NEW):
- View all registered devices from backend (GET /devices)
- Mocopi connection status
- Total sensor count
- Refresh button
---
๐ง Implementation Plan
### Phase 1: Remove Visualizations (1-2 hours)
Goal: Strip out all visualization code
Tasks:
1. Delete `PerformanceView.swift`
2. Delete `VisualizationService.swift`
3. Remove WebSocket `/visualization` code from `NetworkService.swift`
4. Simplify `MainView.swift` tabs
Files to modify:
- `Views/MainView.swift` - Remove PerformanceView tab
- `Services/NetworkService.swift` - Remove WebSocket code
- Delete `Views/PerformanceView.swift`
- Delete `Services/VisualizationService.swift`
Deliverable: App compiles, visualizations gone, sensor streaming still works
---
### Phase 2: Add Strudel Integration (4-6 hours)
Goal: Embed Strudel music engine using WKWebView
Tasks:
1. Create Strudel HTML bundle (1 hour):
- Create `strudel-bundle.html` with:
- Strudel JavaScript library (or MotionStrudel variant)
- Tone.js for Web Audio
- WebSocket client for latent state
- Pattern edit interpreter
- Base musical templates (House, Techno, Ambient)
2. Add WKWebView to app (1 hour):
- Create `StrudelView.swift` - SwiftUI view with WKWebView
- Load `strudel-bundle.html` from app bundle
- Handle Web Audio unlock (user gesture required)
3. Create Swift โ JS bridge (2 hours):
- Create `StrudelBridge.swift`:
- `startMusic()` - unlock audio context, start Strudel
- `stopMusic()` - stop Strudel engine
- `setVolume(level: Float)` - control audio
- `sendLatentState(data: Data)` - forward latent to Strudel
- `sendPatternEdit(edit: String)` - forward edits to Strudel
- Use WKWebView `evaluateJavaScript()` and `postMessage()`
4. Add WebSocket client (1-2 hours):
- Create `LatentStreamService.swift`:
- WebSocket connection to cc-mcs Cloud Run
- Subscribe to latent state stream (50 Hz)
- Subscribe to pattern edit stream (per bar)
- Forward latent โ Strudel via `StrudelBridge`
- Forward edits โ Strudel via `StrudelBridge`
Files to create:
- `Resources/strudel-bundle.html` - Strudel JavaScript engine
- `Views/StrudelView.swift` - WKWebView container
- `Services/StrudelBridge.swift` - Swift โ JS communication
- `Services/LatentStreamService.swift` - WebSocket client
Files to modify:
- `Info.plist` - Add WKWebView permissions
- `Views/MainView.swift` - Add Perform tab with StrudelView
Deliverable: Tap "Start Performance" โ sensors stream, music plays locally
---
### Phase 3: Create Perform UI (2-3 hours)
Goal: Build clean, dancer-focused performance interface
Tasks:
1. Create `PerformView.swift` (new):
- Large "Start/Stop Performance" button
- Audio level meter (read from Strudel via bridge)
- Sensor status grid (iPhone, Watch, AirPods, Mocopi)
- Backend connection indicator
- Stats (collapsible)
2. Add multi-sensor status component:
- Create `MultiSensorStatus.swift`:
- Fetch devices from GET /devices
- Show checkmarks for active sensors
- Group by type (iPhones, Mocopi, Watch, AirPods)
3. Wire up state management:
- Create `PerformViewModel.swift`:
- `isPerforming: Bool`
- `musicPlaying: Bool`
- `activeSensors: [DeviceInfo]`
- `backendConnected: Bool`
- Start/stop logic
Files to create:
- `Views/PerformView.swift` - Main performance UI
- `Views/Components/MultiSensorStatus.swift` - Sensor grid
- `ViewModels/PerformViewModel.swift` - State management
Deliverable: Clean UI, all status indicators working
---
### Phase 4: Simplify Record + Settings (1-2 hours)
Goal: Streamline existing features
Tasks:
1. Simplify `RecordView.swift`:
- Remove complex UI
- Keep essentials: name, start/stop, markers
2. Enhance `SettingsView.swift`:
- Add "Strudel Configuration" section
- Add "Sensor Ecosystem" section (shows all devices)
- Add Mocopi status indicator
Files to modify:
- `Views/RecordView.swift` - Simplify
- `Views/SettingsView.swift` - Add sections
Deliverable: Settings shows Strudel config + ecosystem status
---
### Phase 5: Testing (2-3 hours)
Goal: Verify everything works end-to-end
Tasks:
1. Test sensor streaming:
- Start app, verify POST /api/data works
- Check cc-mcs backend logs for sensor data
2. Test Strudel playback:
- Tap "Start Performance"
- Verify Web Audio unlocks
- Verify music plays from device speakers
3. Test latent streaming:
- Move phone/body
- Verify music reacts to motion
- Check audio level meter responds
4. Test pattern edits:
- Perform large movements (trigger builds/drops)
- Verify musical structure changes
5. Test multi-sensor awareness:
- Run Mocopi bridge (test mode)
- Verify app shows all 10 sensors
6. Test background mode:
- Lock screen while performing
- Verify music continues playing
- Verify sensors keep streaming
Deliverable: Fully working performance app
---
๐ Before/After Comparison
| Aspect | Before | After |
|---|---|---|
| Purpose | Data collection + visualization | Performance app (sensors + music) |
| UI Tabs | Stream, Record, Performance, Settings | Perform, Record, Settings |
| Music | None (use cc-dashboard) | Strudel embedded (local playback) |
| Visualization | Heavy (latent orb, trajectory) | Minimal (audio meter, status) |
| Code Size | ~3,000 lines | ~2,500 lines (Strudel adds ~500, removes ~1,000) |
| Dependencies | SwiftUI + CoreMotion + AVFoundation | SwiftUI + CoreMotion + WKWebView + WebSocket |
| User Flow | Start streaming โ open browser for music | Start performance โ music plays locally |
| Sensor Support | iPhone + Watch + AirPods | iPhone + Watch + AirPods + Mocopi-aware |
| Backend | POST /api/data only | POST /api/data + WS latent stream |
| Primary User | Data collectors | Dancers/Performers |
---
๐ฏ Three-App Ecosystem Strategy (Confirmed)
### App 1: EchelonCapture (This App)
Purpose: Live performance - sensors + music generation
Users: Dancers, performers
Sensors: iPhone, Watch, AirPods (complements Mocopi)
Music: Strudel (embedded WKWebView) + optional Neural Audio Server
Backend: cc-mcs Cloud Run (LIM-RPS + Computational Rehearsal)
Output: Real-time AI-generated music played locally
Keep as separate app because: Primary performance tool, different UX (dancer-focused), music generation core feature
---
### App 2: cc-handguard
Purpose: Behavioral intervention (nail-biting detection)
Users: People with nail-biting habit
Sensors: Apple Watch (wrist motion)
Music: None
Backend: cc-mcs Cloud Run (same backend!)
Output: Haptic intervention when nail-biting detected
Keep as separate app because: Completely different UX (health/behavior), no music, Watch-first sensor strategy
---
### App 3: TrajectoryOS
Purpose: Life trajectory optimization
Users: People optimizing personal growth
Sensors: None (uses backend state)
Music: None
Backend: TrajectoryCore (different backend!)
Output: AI recommendations for life actions
Keep as separate app because: Different domain (life optimization vs motion), different backend, no sensors, no music
---
๐ผ Where Does Music Generation Live?
Answer: Two places, serving different users:
1. EchelonCapture (iOS): Embedded Strudel for dancers
- WKWebView running Strudel JavaScript
- Local audio playback (iPhone speakers/headphones)
- Simplified UI (just performance controls)
- For: Dancers who want to hear music while moving
2. cc-dashboard (Web): Full Strudel + visualization for DJs/producers
- Browser-based Strudel engine
- Desktop/laptop speakers (better for clubs)
- Full visualization (trajectory, latent state, conductor decisions)
- Producer controls (section overrides, style changes, neural audio toggle)
- For: DJs, producers, researchers, live shows
Both can run simultaneously:
- Dancer uses EchelonCapture on iPhone (hears music in headphones)
- DJ uses cc-dashboard on laptop (controls music for audience)
- Both receive same latent state from cc-mcs backend
- DJ can override or augment dancer's music generation
---
โ Decision Matrix: Where Should Strudel Run?
### Option A: Browser only (cc-dashboard) โ
Pros: Single codebase, easier maintenance
Cons:
- Dancers need separate device for music (phone for sensors + laptop for audio = awkward)
- Not mobile-friendly for performers
- Requires WiFi/tethering for audio playback
### Option B: WKWebView in EchelonCapture โ
(RECOMMENDED)
Pros:
- Dancers hear music on-device (iPhone speakers/headphones)
- Native app experience (App Store, background mode)
- Same Strudel JavaScript codebase (shared with cc-dashboard)
- Can still use cc-dashboard for DJ/producer view
Cons:
- WKWebView adds complexity
- Need Swift โ JS bridge
- iOS Web Audio has unlock requirement (minor)
### Option C: Native Swift reimplementation โ
Pros: Maximum control, no Web Audio limitations
Cons:
- Massive effort (rebuild entire pattern engine in Swift)
- Duplicate codebase (Swift + JavaScript)
- Ongoing maintenance burden
- Lose Strudel's declarative pattern language
DECISION: Option B (WKWebView in EchelonCapture) โ
---
๐ Implementation Timeline
### Week 1: Core Restructuring
- [ ] Phase 1: Remove visualizations (1-2h)
- [ ] Phase 2: Add Strudel integration (4-6h)
- [ ] Phase 3: Create Perform UI (2-3h)
- [ ] Phase 4: Simplify Record + Settings (1-2h)
- [ ] Phase 5: Testing (2-3h)
Total: ~10-16 hours (2-3 days)
### Post-Mocopi Hardware Arrival
- [ ] Test with Mocopi bridge running
- [ ] Verify 10-sensor ecosystem (2 iPhone + 1 Watch + 1 AirPods + 6 Mocopi)
- [ ] Test Computational Rehearsal โ Strudel pattern edits
- [ ] Test with Neural Audio Server (GPU phrases)
---
๐ Files to Modify/Create
### Delete
- `EchelonCapture/Views/PerformanceView.swift`
- `EchelonCapture/Services/VisualizationService.swift`
### Create New
- `EchelonCapture/Resources/strudel-bundle.html` - Strudel JS engine
- `EchelonCapture/Views/StrudelView.swift` - WKWebView container
- `EchelonCapture/Views/PerformView.swift` - Main performance UI
- `EchelonCapture/Services/StrudelBridge.swift` - Swift โ JS bridge
- `EchelonCapture/Services/LatentStreamService.swift` - WebSocket client
- `EchelonCapture/Views/Components/MultiSensorStatus.swift` - Sensor grid
- `EchelonCapture/ViewModels/PerformViewModel.swift` - State management
- `EchelonCapture/Models/LatentState.swift` - 25D latent data model
- `EchelonCapture/Models/PatternEdit.swift` - Pattern edit commands
### Modify
- `EchelonCapture/Views/MainView.swift` - Add Perform tab
- `EchelonCapture/Views/RecordView.swift` - Simplify
- `EchelonCapture/Views/SettingsView.swift` - Add Strudel config
- `EchelonCapture/Services/NetworkService.swift` - Keep HTTP POST, remove WS visualization
- `EchelonCapture/Info.plist` - Add WKWebView permissions, background audio
---
๐ฏ Success Criteria
### EchelonCapture is successfully restructured when:
1. โ
All visualization code removed (PerformanceView, VisualizationService)
2. โ
Strudel music engine embedded via WKWebView
3. โ
"Start Performance" button starts sensors + music
4. โ
Music plays locally from iPhone speakers/headphones
5. โ
Music reacts to body movement in real-time
6. โ
Pattern edits from Conductor change musical structure
7. โ
App shows all active sensors (iPhone + Mocopi + Watch + AirPods)
8. โ
Background mode works (screen lock doesn't stop music)
9. โ
No errors, compiles cleanly
10. โ
Works with cc-mcs backend (POST /api/data, WS latent stream)
---
๐ญ Computational Rehearsal Integration
How It Works in EchelonCapture
Data Flow:
1. iPhone sensors (100Hz) โ POST /api/data โ cc-mcs backend
2. cc-mcs: LIM-RPS โ 25D latent state z(t) (50Hz)
3. cc-mcs: Computational Rehearsal โ predicted future z(t+ฮ)
4. cc-mcs: Conductor โ pattern edits based on predicted trajectory
5. cc-mcs โ WebSocket โ EchelonCapture (latent + edits)
6. EchelonCapture: StrudelBridge โ forward to Strudel (WKWebView)
7. Strudel: Apply edits to pattern graph, modulate synth params
8. Web Audio: Output music to iPhone speakers
9. Dancer hears music โ adjusts movement (loop continues)Key Concepts:
- Latent State (25D): tension, excitement, convergence, divergence, energy, velocity, etc.
- Rehearsal: System predicts where your body is going (1-2 seconds ahead)
- Pattern Edits: Symbolic commands like "increase hi-hat density", "drop bass", "add fill"
- Conductor: Musical state machine (Intro/Groove/Build/Climax/Breakdown/Outro)
Strudel receives:
- Fast stream (50Hz): Current latent values โ modulate continuous params (filter, gain, swing)
- Slow stream (per bar): Pattern edits โ change musical structure at beat boundaries
---
๐ Related Documentation
- [MOCOPI_README.md](MOCOPI_README.md) - Mocopi integration overview
- [MOCOPI_ARCHITECTURE_DIAGRAM.md](MOCOPI_ARCHITECTURE_DIAGRAM.md) - System architecture
- [apps/ios/EchelonCapture/cc-rehearsal.md](apps/ios/EchelonCapture/cc-rehearsal.md) - Computational Rehearsal + Strudel integration theory
- [apps/ios/EchelonCapture/README.md](apps/ios/EchelonCapture/README.md) - Current app docs
- [apps/ios/cc-handguard/README.md](apps/ios/cc-handguard/README.md) - HandGuard app
- [apps/ios/TrajectoryOS/README.md](apps/ios/TrajectoryOS/README.md) - TrajectoryOS app
---
๐ค Open Questions for User
1. Strudel HTML Bundle: Should we create a custom `strudel-bundle.html` or use existing Strudel library + custom wrapper?
- Recommendation: Start with minimal custom bundle (Strudel + Tone.js + WebSocket client)
2. Music Styles: What musical templates should we include? (House, Techno, Ambient, etc.)
- Recommendation: Start with House (4/4, kick/hats/bass/pads), add more later
3. Neural Audio Server: Should EchelonCapture support GPU-generated phrases, or Strudel only at first?
- Recommendation: Strudel only initially, add Neural Audio later (Phase 2)
4. Recording Features: Keep local recording, or is EchelonCapture purely for live performance?
- Recommendation: Keep recording for training data collection (useful for Computational Rehearsal)
5. Dual Deployment: Should cc-dashboard also embed Strudel for DJ use?
- Recommendation: Yes - same Strudel codebase, different UI (producer controls vs dancer controls)
---
๐ญ Summary
Key Insight: With Strudel music generation and Mocopi sensors, EchelonCapture transforms from a data collection tool into a complete performance app.
New Role: Dancer-focused app that:
- Streams iPhone/Watch/AirPods sensor data
- Plays AI-generated music locally (Strudel in WKWebView)
- Reacts to your movement via Computational Rehearsal
- Shows multi-sensor ecosystem status (Mocopi awareness)
- No visualizations (simplified UI)
- Real-time musical performance (body โ music in <300ms)
Architecture:
- Sensors: Native Swift (CoreMotion, WatchConnectivity)
- Music: Strudel JavaScript (WKWebView + Web Audio)
- Brain: cc-mcs Cloud Run (Rust LIM-RPS + Computational Rehearsal)
- Communication: HTTP POST (sensors) + WebSocket (latent + edits)
Three-App Strategy: Keep apps separate by domain:
- EchelonCapture: Performance (sensors + music)
- cc-handguard: Behavioral intervention
- TrajectoryOS: Life optimization
Strudel Deployment:
- EchelonCapture (iOS): Embedded WKWebView for dancers
- cc-dashboard (Web): Full browser version for DJs/producers
Next Step: Begin Phase 1 (Remove Visualizations) and Phase 2 (Add Strudel Integration) once user approves plan.
---
Version: 2.0
Status: โ
Ready for review (updated with Strudel integration)
Estimated Work: 10-16 hours (2-3 days)
---
Built for the multi-sensor Computational Choreography ecosystem with Strudel music generation ๐ญ๐ต
Promotion Decision
Attach run IDs, datasets, metrics, and reproduction commands.
Source Anchor
projects/Documentation/_archive/2024-12/ECHELON_CAPTURE_RESTRUCTURING.md
Detected Structure
Method ยท Evaluation ยท Code Anchors ยท Architecture