DJ Agent Completion Plan

# DJ Agent Completion Plan
Linear path from current state to fully integrated motion-driven DJ system

---

## Current State
✅ Python runtime engine running (100 FPS, 1.4ms latency)
✅ DELL equilibrium solver operational
✅ DJ Agent code complete (action space, scheduler, policies, bridge)
⚠️ DJ Agent disabled by default
⚠️ Serato integration not tested
⚠️ Tauri frontend exists but not connected to DJ Agent

---

## Goal State
✅ Motion sensors → DELL → DJ Agent → Serato (verified working)
✅ Tauri UI displays DJ Agent status in real-time
✅ User can toggle tiers, change modes, monitor actions
✅ Full pipeline documented and launchable with one script

---

Linear Completion Plan

### PHASE 1: Basic Serato Connection (Keyboard Mode)
Goal: Verify Python can send keystrokes to Serato
Time: 30 minutes

#### Tasks:
1. Enable DJ Agent in config
- File: `computational-studio/studio/configs/session.yaml`
- Add DJ Agent section with keyboard mode
- Set `tiers_enabled: [0]` (Transport only)

2. Create keystroke test script
- File: `scripts/test_serato_connection.py`
- Send single keystroke (e.g., SPACE for PLAY/PAUSE)
- Verify `keyboard` module works on your OS

3. USER ACTION: Test keystroke
- Grant Accessibility permission (macOS only)
- Open Serato DJ Pro or Serato Play
- Run test script
- Verify keystroke triggers Serato action

Success Criteria:
- ✅ Test script sends keystroke without errors
- ✅ Serato responds to keystroke
- ✅ Permission issues resolved

Rollback: If keyboard mode fails, skip to Phase 2 (MIDI mode)

---

### PHASE 2: MIDI Integration
Goal: Professional MIDI connection to Serato
Time: 45 minutes

#### Tasks:
4. Set up virtual MIDI device
- macOS: Enable IAC Driver in Audio MIDI Setup
- Windows: Install loopMIDI
- Linux: Use ALSA virtual MIDI
- Verify device appears in `mido.get_output_names()`

5. Update config for MIDI mode
- File: `computational-studio/studio/configs/dj.yaml`
- Set `mode: midi`
- Set `midi_port: "IAC Driver Bus 1"` (or loopMIDI port)

6. Create MIDI test script
- File: `scripts/test_midi_connection.py`
- Send test MIDI notes (C1, D1, E1)
- Verify `mido` connects to virtual port

7. USER ACTION: Configure Serato MIDI Learn
- Open Serato → Setup → MIDI → Enable your virtual device
- Map 5 Tier 0 actions:
- PLAY_A → Note C1 (36)
- PLAY_B → Note C#1 (37)
- SYNC_A → Note D1 (38)
- SYNC_B → Note D#1 (39)
- PITCH_NUDGE_UP_A → CC 1

Success Criteria:
- ✅ Virtual MIDI device visible to both Python and Serato
- ✅ Test script sends MIDI without errors
- ✅ Serato responds to MIDI notes
- ✅ 5 actions mapped and verified

Dependencies: Phase 1 complete (or skipped)

---

### PHASE 3: DJ Agent Live Testing
Goal: Full motion → Serato pipeline working
Time: 1 hour

#### Tasks:
8. USER ACTION: Test DJ Agent with Serato
- Start Serato with a loaded track
- Run: `python scripts/run_studio.py`
- Move (simulate sensor input or use real sensors)
- Verify actions trigger on beats

9. Add DJ Agent telemetry
- File: `computational-studio/studio/runtime/engine.py`
- Extend `_serialize_telemetry()` to include:

     'dj_agent': {
         'enabled': self.dj_enabled,
         'last_action': action_name,
         'cooldowns': {tier: remaining_ms},
         'phase_deg': psi * 180 / π,
         'actions_fired_count': total_count
     }

Success Criteria:
- ✅ Engine runs without errors with DJ Agent enabled
- ✅ Actions fire on beat (within quantization window)
- ✅ Cooldowns prevent spam
- ✅ Safety masks prevent invalid actions
- ✅ Telemetry emitted via WebSocket

Rollback: If engine crashes, check logs, disable problematic tiers

---

### PHASE 4: Tauri Frontend Integration
Goal: Visual monitoring and control of DJ Agent
Time: 2 hours

#### Tasks:
10. Update Tauri telemetry types
- File: `computational-studio/apps/desktop/src/telemetry.ts`
- Add DJ Agent fields to `TelemetryFrame` interface

11. Create DJ Agent UI component
- File: `computational-studio/apps/desktop/src/components/DJAgentPanel.tsx`
- Display:
- Enabled/disabled status
- Current mode (keyboard/MIDI)
- Last action + timestamp
- Active cooldowns (progress bars)
- Phase wheel (visual beat indicator)
- Actions fired count
- Safety violations (if any)

12. Integrate into main App
- File: `computational-studio/apps/desktop/src/App.tsx`
- Add `<DJAgentPanel frame={frame?.dj_agent} />`
- Position below main metrics

Success Criteria:
- ✅ TypeScript compiles without errors
- ✅ DJ Agent panel renders
- ✅ Real-time updates from Python telemetry
- ✅ UI shows actions as they fire

Dependencies: Phase 3 complete (telemetry working)

---

### PHASE 5: Interactive Controls
Goal: Control DJ Agent from Tauri UI
Time: 1.5 hours

#### Tasks:
13. USER ACTION: Build and test Tauri app
- Run: `cd computational-studio/apps/desktop && npm run tauri dev`
- Verify app launches
- Verify telemetry connection
- Check DJ Agent panel displays correctly

14. Add DJ Agent Tauri commands
- File: `computational-studio/apps/desktop/src-tauri/src/commands.rs`
- Add commands:

      #[tauri::command]
      async fn dj_enable_tier(tier: u8) -> Result<(), String>

      #[tauri::command]
      async fn dj_disable_tier(tier: u8) -> Result<(), String>

      #[tauri::command]
      async fn dj_set_mode(mode: String) -> Result<(), String>

- Send commands to Python via WebSocket

• File: `computational-studio/studio/runtime/engine.py`
• Add command receiver to handle Tauri requests

• File: `computational-studio/apps/desktop/src/components/DJAgentPanel.tsx`
• Add buttons/toggles for tier enable/disable
• Add mode switcher (keyboard/MIDI)

Success Criteria:
- ✅ Tauri commands send to Python without errors
- ✅ Python engine responds to commands
- ✅ UI reflects state changes immediately
- ✅ Tiers can be enabled/disabled live

Dependencies: Phase 4 complete (UI working)

---

### PHASE 6: Full Integration & Launch
Goal: One-command launch of entire system
Time: 1 hour

#### Tasks:
15. End-to-end integration test
- USER ACTION: Full pipeline test
1. Start Serato DJ Pro with track loaded
2. Start Python engine: `python scripts/run_studio.py`
3. Start Tauri app: `npm run tauri dev`
4. Connect motion sensors (or simulate)
5. Move → verify actions in Serato
6. Check Tauri UI updates in real-time
7. Toggle tiers from UI → verify effect

16. Create unified launch script
- File: `scripts/launch_full_studio.sh` (macOS/Linux)
- File: `scripts/launch_full_studio.bat` (Windows)
- Starts:
1. SuperCollider server (if using audio)
2. Python runtime engine (background)
3. Tauri desktop app (foreground)
- Checks:
- Virtual MIDI device exists
- Required ports available
- Config files valid
- Cleanup on exit (graceful shutdown)

Success Criteria:
- ✅ Single script launches entire system
- ✅ All components connect automatically
- ✅ Motion → DELL → DJ Agent → Serato → UI (full loop)
- ✅ Latency <10ms maintained with all components
- ✅ No crashes over 5-minute test run

Dependencies: Phases 1-5 complete

---

### PHASE 7: Documentation & Demo
Goal: Shareable, reproducible workflow
Time: 1 hour

#### Tasks:
17. Document complete workflow
- Update: `README_DJ_AGENT.md`
- Add sections:
- Prerequisites (software, permissions, hardware)
- Quick Start (one-script launch)
- Troubleshooting (common errors)
- Architecture diagram (full system)
- Performance tuning
- Next steps (training, new tiers)

18. USER ACTION: Create demo video
- Record 2-minute screen capture:
1. Launch system
2. Show Tauri UI with metrics
3. Move (sensor input)
4. Show actions triggering in Serato
5. Toggle tier in UI
6. Show different action types
- Export as GIF or MP4
- Add to README

Success Criteria:
- ✅ Documentation clear enough for new user to set up in 30 minutes
- ✅ Demo video shows full pipeline working
- ✅ Troubleshooting section covers known issues

Dependencies: Phase 6 complete

---

Dependency Graph

PHASE 1 (Keyboard)
    ↓
PHASE 2 (MIDI) ← parallel to Phase 1, but MIDI preferred
    ↓
PHASE 3 (Live Testing + Telemetry)
    ↓
PHASE 4 (Tauri UI - Display Only)
    ↓
PHASE 5 (Tauri UI - Interactive Controls)
    ↓
PHASE 6 (Full Integration + Launch Script)
    ↓
PHASE 7 (Documentation + Demo)

---

Time Estimates

---

Risk Mitigation

### Known Risks:
1. Keyboard module fails on macOS Accessibility
- Mitigation: Skip to MIDI mode (Phase 2)

2. MIDI device not visible in Serato
- Mitigation: Use keyboard mode, or check IAC Driver settings

3. Latency budget exceeded with DJ Agent enabled
- Mitigation: Reduce solver iterations, disable telemetry, use faster DJ action checking

4. Tauri build fails
- Mitigation: Pre-built binaries, or use Python-only CLI mode

5. Serato doesn't respond to MIDI
- Mitigation: Check MIDI Learn mappings, verify virtual device active

---

Success Metrics

### MVP Success (Phases 1-3):
- Motion → Serato actions working
- Beat quantization accurate (±15°)
- No safety violations
- Stable over 5-minute run

### Full Success (Phases 1-7):
- One-command launch
- Real-time UI monitoring
- Interactive tier control
- Documented and demoed
- <10ms end-to-end latency maintained

---

Next Steps After Completion

1. Record training data (Task 25 from previous TODO)
- Log sessions with DJ Agent enabled
- Capture motion + actions for imitation learning

2. Train imitation model (Task 26)
- Learn from recorded sessions
- Predict next action from motion/phase

3. Unlock higher tiers
- Tier 4: Library & load
- Tier 5: Blend & structure

4. Hardware controller integration
- Read physical deck inputs
- Hybrid human/AI control

5. Production performance
- Full live set with motion-driven mixing
- Multi-deck control
- Generative audio stems

---

Starting Point: Phase 1, Task 1

Ready to begin? Start with:

# Enable DJ Agent in config
# Update: computational-studio/studio/configs/session.yaml

Let me know when you're ready to start Phase 1!