Computational Choreography — Architecture Design V1

P1[C:visual_experience | A:sensor_pipeline | B:sound_design | C:training_arch] -> GATE

┌─────────────────────────┐
│   MotionMix        9:16 │ ← italic serif watermark
│                         │
│    ┌───────────────┐    │
│    │               │    │
│    │  YOUR CHEST   │    │
│    │  (cropped)    │    │
│    │               │    │
│    │  HOUSE        │    │ ← genre + BPM overlay
│    │  134 BPM      │    │
│    └───────────────┘    │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─  │ ← thin divider
│    ┌───────────────┐    │
│    │               │    │
│    │   ● ORB ●     │    │ ← reactive to movement
│    │  /|||||||||\   │    │    spikes = energy
│    │               │    │    color = genre
│    └───────────────┘    │
│                         │
│  ● REC 02:34            │ ← recording indicator
└─────────────────────────┘

┌─────────────────────────┐
│  MotionMix    3 📷  9:16│ ← camera count badge
│                         │
│  ┌─────────────────┐    │
│  │  FRONT / SIDE / │    │ ← AutoDirector switches
│  │  OVERHEAD       │    │    based on Echelon state
│  │  (auto-cut)     │    │
│  │                 │    │
│  │  TECHNO         │    │
│  │  132 BPM        │    │
│  └─────────────────┘    │
│  ┌──┐ ┌──┐ ┌──┐        │ ← thumbnail strip of all angles
│  │F │ │S │ │O │        │    highlighted = active
│  └──┘ └──┘ └──┘        │
│  ─ ─ ─ ─ ─ ─ ─ ─ ─ ─  │
│    ┌───────────────┐    │
│    │   ● ORB ●     │    │
│    └───────────────┘    │
└─────────────────────────┘

                     ┌─────────────┐
iPhone 2 ──MCPeer──► │             │
iPhone 3 ──MCPeer──► │  iPhone 1   │──► Capture Server :9405
iPad 1   ──MCPeer──► │  (Director) │         │
iPad 2   ──MCPeer──► │             │         │
                     └─────┬───────┘         │
                           │                 │
Watch    ──WCSession──► iPhone 1             │
Mocopi   ──UDP :12351──► mocopi-bridge.py ──►│
MacBook  ──WebSocket──► webcam-relay ───────►│
                                             │
                                    ┌────────▼────────┐
                                    │ Capture Server   │
                                    │ :9405            │
                                    │                  │
                                    │ Records ALL as   │
                                    │ PhraseData       │
                                    └────────┬─────────┘
                                             │
                                    ┌────────▼────────┐
                                    │ Session Output   │
                                    └─────────────────┘

sessions/2026-03-23_morning/
├── capture.json            # Session metadata, device list, angles
├── echelon_latent.npy      # (T, 16) Echelon brain latent trace
├── echelon_sections.npy    # (T,) Section states 0-6
├── motion_25d.npy          # (T, 25) Encoded 25D motion
├── clip_embeddings.npy     # (T/4, 512) CLIP embeddings every 0.5s
├── clip_clusters.npy       # (T/4,) Cluster labels
├── pose_front.npy          # (T, 14, 2) Vision BodyPose from front camera
├── pose_side.npy           # (T, 14, 2) Vision BodyPose from side camera (if available)
├── mocopi_skeleton.npy     # (T, 27, 7) Mocopi quaternion+position per bone (if available)
├── watch_imu.npy           # (T, 6) Watch accel+gyro (if available)
├── audio.npz               # mel, chroma, mfcc, rms, centroid, onset
├── video_front.mov         # Cropped vertical video from front camera
├── video_composite.mov     # AutoDirector composite (multi-angle edit)
└── metadata.json           # tempo, duration, participant, genre, devices used

Layer 1: PERCUSSION (existing)
  kick, clap, hat, bass patterns
  Driven by: energy state (base/build/drop/chill)
  24 patterns across 6 genres

Layer 2: HARMONIC (built)
  Chord progressions (3 per genre = 18 total)
  Bass follows chord root
  Driven by: CLIP cluster + tension scalar

Layer 3: TEXTURAL (to build)
  Granular clouds, noise sweeps, spectral freezes
  Driven by: movement QUALITY (smooth=warm grain, sharp=digital glitch)
  Uses Tone.js GrainPlayer or custom noise generators

Layer 4: STRUCTURAL (to build)
  Pattern transformations applied at phrase boundaries
  reverse, halftime, doubletime, swing, striate
  Driven by: movement TRANSITIONS (cluster change = pattern transform)

Section 0 (Entry)       → Cluster 2 sound: minimal, building
Section 1 (MicroInit)   → Cluster 2→1 transition: layers adding
Section 2 (Stable)      → Cluster 0 sound: full groove
Section 3 (Divergence)  → Pattern transforms activate: variety
Section 4 (Transitional)→ Cluster 1 sound: harmonic focus, tension
Section 5 (Reformation) → Cluster 0 peak: all layers, maximum energy
Section 6 (Resolution)  → Cluster 2 return: wind down, minimal

Daily Session Video (9:16 vertical, 5-30 min)
         │
         ├─► Extract frames (2fps, chest-cropped 224×224)
         │
         ├─► CLIP ViT-B/32 encode → 512-dim vectors
         │
         ├─► KMeans cluster → movement vocabulary labels
         │
         ├─► Pair with audio (Encodec tokenize → 128-dim continuous)
         │
         └─► Training samples:
             {
               clip_embedding: (512,),        # What the body looks like
               motion_25d: (T_window, 25),    # Echelon-encoded trajectory
               audio_tokens: (128, L),        # Encodec continuous audio
               cluster_label: int,            # Which movement vocabulary
               anticipation_scalars: (7,),    # Geometric scalars (when available)
             }

Week 1-2: Collect 15+ sessions
         │
         ├─► CLIP embeddings accumulated
         │   Clusters refine (3 → 5 → 8 as vocabulary grows)
         │
         ├─► CC-MotionGen V2 trains on (audio → motion)
         │   MotionDiT-Tiny (1.8M params) on Mac4+5
         │   "Given this house track, predict Mohamed's body trajectory"
         │
         └─► Chest flex model retrains with more angles + contexts

Week 3-4: 30+ sessions accumulated
         │
         ├─► EchelonDiT trains on (motion → audio)
         │   43M params, Encodec codec
         │   "Given Mohamed's movement, generate matching audio"
         │
         ├─► Movement vocabulary classifiers trained per cluster
         │   One MobileNetV2 per movement type (auto-discovered from CLIP)
         │
         └─► Rehearsal Engine upgrades:
             Linear extrapolation → CC-MotionGen prediction
             "Where will Mohamed be in 2 seconds?"

Month 2+: 60+ sessions
         │
         ├─► EchelonDiT replaces Strudel as primary audio engine
         │   Neural audio that sounds like YOUR music taste
         │
         ├─► CLIP embeddings as EchelonDiT conditioning
         │   Instead of 25D motion: 512D CLIP vectors
         │   Richer representation of body state
         │
         └─► Rehearsal Engine with trained MotionGen
             Music arrives BEFORE your body
             The system anticipates, you respond
             THE LOOP CLOSES

Day N: You move → System records → Models train overnight
                                          │
Day N+1: Models predict better → Music responds faster
         │                                │
         └─► You respond to better music → New movement patterns emerge
                                                    │
Day N+2: Models learn your RESPONSE patterns ◄──────┘
         The system learns how you respond
         to being musically led
                                    │
Day N+3: Rehearsal Engine predicts your response
         to its own predictions
         │
         └─► EMERGENT CHOREOGRAPHY
             Neither you nor the system "lead"
             The movement-music relationship
             evolves as its own entity

YOUR BODY
    │
    ├── Mocopi (27 bones) ──UDP──┐
    ├── Watch (wrist IMU) ──WS──┤
    ├── iPhones (camera×3) ─────┤
    ├── iPads (camera×2) ───────┤    ┌──────────────────┐
    └── MacBook (webcam) ───────┼──► │ CAPTURE SERVER    │
                                │    │ :9405             │
                                │    │                   │
                                │    │ Records:          │
                                │    │ - 25D motion      │
                                │    │ - CLIP embeddings  │
                                │    │ - Echelon latent  │
                                │    │ - Audio           │
                                │    │ - Multi-angle vid │
                                │    └────────┬──────────┘
                                │             │
              ┌─────────────────┤             │
              ▼                 │             ▼
    ┌──────────────┐            │   ┌──────────────────┐
    │ ECHELON BRAIN│            │   │ TRAINING PIPELINE │
    │ (Rust, <5ms) │            │   │                   │
    │ 16D latent   │            │   │ CC-MotionGen      │
    │ 7 sections   │            │   │ EchelonDiT        │
    │ DELL equilib │            │   │ Movement classif. │
    └──────┬───────┘            │   │ CLIP clustering   │
           │                    │   └────────┬──────────┘
           │ OSC @ 60Hz         │            │
     ┌─────┼──────┐             │   ┌────────▼──────────┐
     ▼     ▼      ▼             │   │ REHEARSAL ENGINE   │
  ┌─────┐┌─────┐┌──────┐       │   │                   │
  │Strud││Orb  ││Touch │       │   │ MotionGen predicts │
  │el   ││Viz  ││Dsgner│       │   │ EchelonDiT pre-gen │
  │Audio││     ││(live)│       │   │ Music LEADS body   │
  └─────┘└─────┘└──────┘       │   └───────────────────┘
     │      │      │            │
     ▼      ▼      ▼            │
  ┌──────────────────────┐      │
  │  RECORDING (ReplayKit)│      │
  │  Split-screen 9:16    │      │
  │  Body + Orb + Overlays│      │
  │  Auto-cut multi-cam   │      │
  └──────────┬────────────┘      │
             │                   │
             ▼                   │
  ┌──────────────────────┐      │
  │  CONTENT PIPELINE     │      │
  │                       │      │
  │  Auto-clip reels      │      │
  │  3/day to @granddiomande│    │
  │  Build→Peak arcs      │      │
  │  Mystery→Reveal series│      │
  └───────────────────────┘      │