HANDOFF: Depth-Reactive Visuals + LUME Hardware Product + Creative Agency

HANDOFF: Depth-Reactive Visuals + LUME Hardware Product + Creative Agency

Date: 2026-04-04
Author: Mohamed Diomande (via Claude Opus)
Recipient: Codex (continuation agent)
Session: b8e3a146 — Full-day session covering three interconnected initiatives

---

TABLE OF CONTENTS

1. [Overview — Three Initiatives](#1-overview)
2. [Initiative 1: Depth-Reactive Interactive Visual System](#2-depth-reactive)
3. [Initiative 2: LUME Hardware Product](#3-lume)
4. [Initiative 3: Diomande Creative Agency](#4-agency)
5. [Infrastructure State](#5-infrastructure)
6. [File Inventory](#6-files)
7. [What's Working](#7-working)
8. [What's Blocked / Incomplete](#8-blocked)
9. [Next Steps (Priority Order)](#9-next-steps)
10. [Reference Material](#10-references)

---

1. OVERVIEW {#1-overview}

This session started from an Instagram reel by Duncan Fewkes showing depth-reactive interactive installations, and evolved into three interconnected projects:

1. Depth-Reactive Visual System — A Unity 6 + TouchDesigner pipeline that uses depth cameras to create real-time interactive particle/fluid visuals driven by body movement and audio
2. LUME — A $1,299 standalone hardware product (Jetson Orin Nano + Orbbec Femto Bolt) that packages the visual system into a wall-mounted device for dance studios, venues, and content creators
3. Diomande Creative — A new media art agency inspired by Hybrid Xperience, Projection Mapping World, and the immersive art sector (teamLab, Mercer Labs, ARTECHOUSE)

All three share the same codebase and technical pipeline. The depth-reactive system IS the software that runs on LUME, and the agency uses it as portfolio work.

---

2. INITIATIVE 1: DEPTH-REACTIVE INTERACTIVE VISUAL SYSTEM {#2-depth-reactive}

### Origin
Reverse-engineered from Duncan Fewkes' Instagram work. His pipeline: Azure Kinect depth camera → GPU fluid simulation → VFX Graph particles + audio reactivity + color cycling. We rebuilt this from scratch for the MotionMix computational choreography mesh.

Architecture

iPhone LiDAR (Record3D, prototype)  OR  Orbbec Femto Bolt (production)
    │
    ▼
DepthSource.cs (abstract provider)
    │
    ├──► DepthProcessor.cs → DepthReprojection.compute
    │       → Point Cloud + Silhouette Mask + Edge Texture
    │
    ├──► OpticalFlowProcessor.cs → OpticalFlow.compute
    │       → Motion Vectors (ping-pong RG16F)
    │
    └──► FluidSimulator.cs → FluidSim.compute (5 kernels)
            → Velocity Field + Density Field (ping-pong textures)
            │
            ├──► AudioAnalyzer.cs (512-bin FFT, 4-band split, transient detection)
            │       → AudioToFluid.cs (energy/bass/transient → fluid params)
            │       → AudioToPalette.cs (transient → palette transition)
            │       → AudioToVFX.cs (FFT bands → particle emission/size/turbulence)
            │
            ├──► ColorPaletteManager.cs (12 palettes, 256x1 GPU texture, 2s crossfade)
            │
            ├──► VFX Graph particles (spawn on silhouette, velocity from fluid)
            │
            └──► Display (HDMI out, fullscreen borderless)

Mesh Integration:
    BridgedClient.cs → WebSocket to bridged :9463
        → MocopiReceiver.cs (28-bone skeleton @ 30Hz)
        → BridgedAudioReceiver.cs (BPM, beat position, bar count)
    EchelonReceiver.cs → WebSocket to hub :9420
        → 104D dynamics (energy, tension, curvature, coherence, complexity)
    ChoreographyOverlay.cs → Echelon dynamics modulate fluid + particles
    PoseTriggers.cs → Mocopi poses trigger visual behavior switches
    SkeletonParticleEmitter.cs → particles at 28 joint positions
    OSCSender.cs → depth motion summary back to mesh via OSC :9462

Compute Shaders (3 files, fully implemented)

DepthReprojection.compute (`Assets/Shaders/Compute/`)
- 2 kernels: `CSMain` (deproject + world transform), `CSBuildSilhouette` (Sobel edge detection)
- Input: raw depth texture + camera intrinsics (focal length, principal point)
- Output: point cloud buffer (float4), silhouette mask (RFloat), edge texture
- Configurable: min/max depth, camera-to-world extrinsic matrix

OpticalFlow.compute
- 2 kernels: `CSComputeGradients` (Ix, Iy, It), `CSIterateSolver` (Jacobi)
- Horn-Schunck dense optical flow between consecutive depth frames
- Ping-pong output: RG16F flow field (motion vectors)
- Configurable: alpha (smoothness, default 50), iterations (default 15)

FluidSim.compute
- 5 kernels: `CSAdvect` (semi-Lagrangian), `CSInjectForces` (motion + audio + silhouette), `CSDivergence`, `CSPressureSolve` (Jacobi), `CSSubtractGradient`
- Full Stable Fluids (Jos Stam) implementation
- 4 ping-pong texture pairs: velocity (RGFloat), density (RFloat), pressure (RFloat), divergence (RFloat)
- Audio parameters: `_AudioEnergy`, `_AudioBass`, `_AudioTransient` (one-shot impulse)
- Configurable: resolution (512), viscosity (0.0001), forceScale (5), densityDecay (0.98), vorticityStrength (0.3), pressureIterations (30)

C# Scripts (24 files)

Core (10 files):
| File | Purpose | Key API |
|------|---------|---------|
| `DepthSource.cs` | Abstract depth provider (Record3D/Femto) | `GetDepthTexture()`, `GetColorTexture()`, intrinsics properties |
| `DepthProcessor.cs` | Dispatches DepthReprojection.compute | `SilhouetteMask`, `ReprojectedDepth`, `PointCloud` |
| `OpticalFlowProcessor.cs` | Dispatches OpticalFlow.compute | `FlowField` (RG16F) |
| `FluidSimulator.cs` | Dispatches all 5 FluidSim passes | `VelocityField`, `DensityField`, `AudioEnergy/Bass/Transient` |
| `IdleDetector.cs` | SuperHot freeze/burst mode | `IsIdle`, `TimeSinceMotion`, 2s timeout, 10x resume burst |
| `PoseTriggers.cs` | Mocopi pose → visual behavior switch | Arms spread=Wing, Arms up=Burst, Crouch=Swirl, 0.5s hold, 2s cooldown |
| `CalibrationTool.cs` | 3-point floor calibration + manual adjust | Arrow keys for position, brackets for rotation, Tab for calibration mode |
| `DisplayManager.cs` | Fullscreen borderless, multi-monitor | F11=fullscreen, Esc=windowed, C=cursor toggle, 60fps target |
| `AttractMode.cs` | Ambient particles when no one present | Breathing sine wave on emission, slow palette cycling, 5s idle → activate |
| `PresetManager.cs` | Save/load venue configurations | Shift+0-9=save, 0-9=load, JSON to `Application.persistentDataPath/Presets/` |
| `FrameRecorder.cs` | Screenshot + sequence capture | P=screenshot, R=record toggle, 30fps PNG sequence, 2x superSize |

Audio (4 files):
| File | Purpose | Key API |
|------|---------|---------|
| `AudioAnalyzer.cs` | Local FFT analysis | `SubBass/Bass/Mid/High`, `TotalEnergy`, `IsTransient`, 512 bins, Hanning window |
| `AudioToFluid.cs` | Wires audio → fluid sim | `AudioEnergy`, `AudioBass`, `IsTransient → AudioTransient = 1.0` |
| `AudioToPalette.cs` | Transient → palette transition | 3s cooldown, optional bass saturation scrub via `_PaletteSaturationBoost` global |
| `AudioToVFX.cs` | FFT bands → VFX Graph properties | `AudioEmissionRate/ParticleSize/Turbulence/Burst/Bass`, smoothing 0.8 |

Visual (4 files):
| File | Purpose | Key API |
|------|---------|---------|
| `ColorPaletteManager.cs` | 12 palettes, GPU-baked gradients | `CurrentPaletteTexture` (256x1 Texture2D), `TriggerTransition()`, `TransitionTo(int)`, 8s auto-cycle, 2s crossfade |
| `ParticleBehaviorSwitcher.cs` | Swap VFX Graph assets | `NextBehavior()`, `SetBehaviorByName(string)`, 15s auto-cycle optional |
| `SkeletonParticleEmitter.cs` | Mocopi joints as particle emitters | 28-joint GraphicsBuffer, per-bone weight (hands 3x, feet 2x, spine 0.5x) |
| `ChoreographyOverlay.cs` | Echelon 104D → visual modulation | Energy→force, Tension→vorticity, Coherence→emission, Complexity→turbulence |

Network (5 files):
| File | Purpose | Key API |
|------|---------|---------|
| `BridgedClient.cs` | WebSocket to bridged :9463 | `OnMotionSkeleton/OnBeatEvent/OnAudioEvent`, exponential backoff (1s→30s), ConcurrentQueue |
| `BridgedAudioReceiver.cs` | Parses beat.* events | `BPM`, `BeatPosition`, `BarCount`, `BeatPhase` (0-1), `OnBeat` (one-frame pulse) |
| `MocopiReceiver.cs` | 28-bone skeleton parser | `JointPositions[28]`, `JointRotations[28]`, `HasSkeleton`, stride=7 floats (xyz + quat) |
| `EchelonReceiver.cs` | 104D dynamics from hub :9420 | `Energy/Tension/Curvature/Coherence/Complexity`, `RawDynamics[104]`, volatile thread-safe |
| `OSCSender.cs` | Depth motion → mesh via OSC :9462 | `/depth/motion` message (energy float, audioEnergy float, idle int), 30Hz UDP |

### 12 Default Color Palettes
Fire, Ocean, Neon Pink, Emerald, Violet Storm, Sunset, Arctic, Gold, Toxic, Blood Moon, Hologram, Monochrome. Each baked to 256x1 RGBA32 texture via `MakeGradient(Color a, b, c, d)`.

TouchDesigner Pipeline (Mac5)

Files deployed to Mac5 (`Desktop/cc-touchdesigner/`):
| File | Purpose |
|------|---------|
| `depth_reactive_builder.py` | Builds 22-node TD network: optical flow → GLSL fluid sim → audio FFT → color palette → particles → bloom |
| `depth_fluid_sim.glsl` | Stable Fluids shader for TD GLSL TOP feedback loop. Semi-Lagrangian advection, force injection from optical flow + silhouette + audio, vorticity confinement, single-pass Jacobi pressure |
| `lidar_to_td.py` | iPhone LiDAR → TCP :9502 bridge. Connects to multicam-server on Mac1 (:9404). Handles full + delta-compressed depth frames. Multi-phone support (front/left/right offsets) |
| `lidar_network_builder.py` | Builds /cc/lidar/ container: TCP DAT, 3 Geometry COMPs (front/left/right), Particle SOP + Trail SOP ghost sculpture, depth color GLSL shader, auto-orbit camera, 1920x1080 render |
| `inject_live_network.py` | Motion signals wiring: Mocopi quaternions → lean/energy/spread/head, wires to geo_orb, geo_aurora, camera, bloom, level, motion trail feedback |
| `cc_network_builder.py` | Base CC Echelon World network |
| `cc-echelon.toe` | Main TouchDesigner project file with /cc container |
| `mocopi_to_td.py` | Mocopi skeleton bridge |

TD depth_reactive_builder.py creates these nodes at /cc/depth_reactive/:

depth_in → silhouette_mask (threshold) → silhouette_edge (Sobel)
         → optical_flow (TD built-in Optical Flow TOP)
         → fluid_sim (GLSL TOP, 512x512 32-bit RGBA, feedback loop)
              ← fluid_uniforms (script CHOP driving GLSL params from audio)
              ← audio_in → audio_spectrum (512 FFT) → audio_bands (4-band + transient)
         → fluid_density (shuffle B channel) → fluid_colored (lookup from color_palette ramp)
         → comp_final (colored fluid + edge glow) → bloom → out1
         + geo_particles (128x128 grid → noise → particle SOP → trail) → render_particles (1920x1080)
         → comp_all (2D fluid + 3D particles) → final_out

Unity Project on Mac4

Location: `[home-path]` on Mac4 ([ip])
Unity Version: 6000.0.72f1 (Unity 6 LTS, ARM64)
Packages (manifest.json): URP 17.0.3, VFX Graph 17.0.3, Input System 1.11.2, Post Processing 3.4.0, Mathematics 1.3.2, Burst 1.8.18, Collections 2.5.1

Status: Project created, all 24 .cs scripts + 3 .compute shaders deployed. Assembly definitions REMOVED (were causing cross-reference errors). Two compile errors FIXED (CalibrationTool.cs SetExtrinsics, FrameRecorder.cs ScreenCapture). One deprecation warning remains (DisplayManager.cs Display.Activate). Project should compile clean.

What's NOT yet done in Unity:
- No scenes created (need main scene with GameObjects wired to scripts)
- No VFX Graph assets created (.vfx files for DepthParticles, Wing, Jellyfish, Swirl behaviors)
- No URP pipeline asset configured
- No Record3D integration (needs Record3D Unity package from Asset Store or GitHub)
- No post-processing stack configured (Bloom, Color Grading, Vignette, Chromatic Aberration)

---

3. INITIATIVE 2: LUME HARDWARE PRODUCT {#3-lume}

### Concept
A $1,299 wall-mounted device that turns any room into an interactive visual studio. Depth camera captures body movement, GPU fluid simulation creates reactive particles, built-in cameras record composited content (body + visual effects), royalty-free audio engine means post anywhere.

Product Name: LUME ("Move in light.")

Physical Design

22" W x 5" H x 3" D (56cm x 12.7cm x 7.6cm)
4.2 lbs (1.9 kg)
Matte black anodized aluminum + soft-touch polycarbonate front
Wall bracket (included) or shelf-stand mode (rubber feet)

Front: Wide cam | Orbbec Femto Bolt (depth 120deg) | Tight/portrait cam
       3-mic array | LED mode ring | Stereo speakers
Rear:  Jetson Orin Nano Super | 512GB NVMe | WiFi 6E
       DC 65W | HDMI 2.1 | USB-C x2 | 3.5mm out | exhaust vents

### Hardware BOM (at 3,000 unit volume)
| Component | Cost |
|-----------|------|
| Orbbec Femto Bolt | $340 |
| Jetson Orin Nano Super | $200 |
| NVMe 512GB | $35 |
| 2x Sony IMX577 4K cameras | $28 |
| 3-mic MEMS array + ADC | $9 |
| 2x 3W speakers | $6 |
| WiFi 6E module | $9 |
| Custom carrier PCB | $22 |
| CNC aluminum enclosure | $32 |
| Thermal (heat pipe + fan) | $5 |
| 65W USB-C PD adapter | $9 |
| Cables, LED ring, mount kit, packaging | $20 |
| TOTAL | $715 |

**Retail $1,299 = 45

### Two Audio Modes
1. LISTEN (default): Mic array captures room audio. FFT drives visuals. Content has room music (copyright depends on source).
2. GENERATE (the moat): Echelon Rust engine synthesizes royalty-free music reactive to movement energy. Content is 100

### Content Pipeline
- Two cameras (wide landscape + tight portrait) record simultaneously
- GPU compositor overlays rendered visual effects onto camera feed in real-time
- AI Director (Murch-scoring from MotionMix) auto-edits:
- 60-second portrait reel (9:16, best moments by energy + composition + beat sync)
- 30-second highlight (1:1, multi-angle cuts)
- WiFi Direct to companion app. Content on phone in <90 seconds. One-tap share.

### Business Model
| Layer | Price | When |
|-------|-------|------|
| Hardware | $1,299 | Day 1 |
| LUME Create | $19/mo | Day 1 — all presets, 4K, AI editor, Suno tracks |
| LUME Studio | $49/mo/device | Day 1 — multi-device, class mode, analytics |
| LUME Venue | $149/mo/device | Day 1 — enterprise, branded, remote mgmt |
| Marketplace | $5-30/pack | Month 6 — visual/audio packs, 70/30 split |
| API/SDK | $99/mo | Month 12 — developer ecosystem |
| LUME Social | Free | Month 24 — TikTok-like feed for LUME content |

### Revenue Projections
| Year | Units | Subs | Hardware | Sub Rev | Marketplace | Total |
|------|-------|------|----------|---------|-------------|-------|
| Y1 | 500 | 350 | $649K | $80K | $0 | $729K |
| Y2 | 3,000 | 2,800 | $3.9M | $806K | $140K | $4.85M |
| Y3 | 15,000 | 12,000 | $19.5M | $3.5M | $1.2M | $24.2M |
| Y5 | 80,000 | 70,000 | $104M | $20.2M | $15M | $139.2M |

### Path to $1B Valuation
- At 15-20x ARR: need $50-67M ARR = 139K-186K subscribers
- Peloton comp: $8.1B at IPO with 511K subs at $39.99/mo
- Timeline: Month 3 iPad app → Month 4 Kickstarter → Month 5 pre-seed $500K → Month 9 first 500 units → Year 3 $24M → Year 5 $139M → $1B+

### Competitive Landscape
| Company | What | Price | LUME's angle |
|---------|------|-------|-------------|
| Augmenta | Tracking server (multi-camera fusion) | ~$10-20K | LUME = self-contained, $1,299 |
| Disguise | Media server (projection/LED) | $20-42K | LUME = all-in-one, no AV crew |
| Mercer Labs | Immersive museum (NYC, $50/visit) | $38-50/visit | LUME = permanent, $1,299 one-time |
| teamLab | Large-scale immersive (2.3M visitors/yr) | $30-40/visit | LUME = personal, creates content |
| ARTECHOUSE | Rotating exhibitions | $25-35/visit | LUME = always on, your content |

### Technical Risks (from Evo3 risk audit)
1. R1 CRITICAL: Unity on Jetson performance. Must validate 60fps with 256x256 fluid + 30K particles + 1080p. Fallback: port shaders to native CUDA/Vulkan.
2. R2 CRITICAL: Simultaneous render + record. Mitigation: Jetson NVENC hardware encoder (independent of GPU compute).
3. R5 MEDIUM: BOM exceeds $700. Mitigation: Orbbec volume pricing, drop speakers V1, 1080p cameras.
4. R6 MEDIUM: FCC/CE certification. Jetson module is pre-certified. Budget $30K for enclosure testing.

### Full Evo3 Output
9 files at `Desktop/evo-cube-output/computational-choreography-device/`:
- `stage0-research.md` (185 lines) — ground truth from codebase + market data
- `stage1-path-a.md` through `stage1-path-f.md` — 6 divergent product visions (Monolith, Bar, Portal, Mirror, Platform, Constellation)
- `stage2-compound.md` (397 lines) — 8-step synthesis into LUME
- `stage3-expand-master-plan.md` (565 lines) — 10 risks, 5 expanded specs, 48 tasks, 12-month roadmap

---

4. INITIATIVE 3: DIOMANDE CREATIVE AGENCY {#4-agency}

### Concept
Full-spectrum creative technology agency. Not an aggregator (like Projection Mapping World). A production studio with actual technical capability, using the depth-reactive visual system as portfolio work.

### Positioning
- PMW = aggregator (features others' work, narrow scope)
- Hybrid Xperience = premium studio (Dubai, Gulf luxury brands)
- Duncan Fewkes = solo creative coder (no agency)
- Diomande Creative = production studio + technology platform + mesh infrastructure

### Inspiration References Ingested
| Source | What | Key Insight |
|--------|------|-------------|
| @duncanfewkes | Depth-reactive installations | Direct technical reference for pipeline |
| @projection_mapping_world | New Media Artist Agency (217K followers) | Curated-feed-as-storefront model |
| @hybridxperience | Dubai immersive studio | "Your reflection, reimagined in color" — same pipeline we built. Clients: F1, Pepsi, NEOM |
| @lazylighting | Projection mapping via mobile app | Accessible, viral content approach (31K likes) |
| @stereo.drift | Computer vision + body tracking + JS | Roboflow + MediaPipe creative coding |
| @augmenta.tech | Tracking server platform | Professional multi-camera fusion, TD/Unity plugins, OSC output |
| Mercer Labs | Immersive museum NYC (36K sqft) | "The Engine" room = particles react to presence = exactly what LUME does. $50/visit. |

### Landing Page
Location: `Desktop/creative-agency-site/index.html`
Status: V2 built with creative director spec. WebGL particle background (18,000 particles, curl noise flow, cursor repulsion with persimmon glow), custom dual-ring magnetic cursor, warm obsidian palette (#0A0908 ground, #FF4D1A persimmon accent), Space Grotesk + IBM Plex Mono typography, mix-blend-mode hero text, scroll reveal animations.
Mohamed's feedback: "Too narrowed and focused on what we are doing rather than the entire vision." Needs broader vision, not just depth-reactive. Should encompass the full creative agency concept.
NOT YET DEPLOYED to Vercel.

### Agency Concept Doc
Full doc at `Desktop/DepthReactiveVisuals/Docs/CREATIVE-AGENCY-CONCEPT.md` with competitive analysis, revenue streams, portfolio pieces, next steps.

---

5. INFRASTRUCTURE STATE {#5-infrastructure}

### Machine Roles (Updated 2026-04-04)
| Machine | IP | Role | What's on it |
|---------|-----|------|-------------|
| Mac1 | [ip] | Orchestrator | bridged :9460, NATS :4222, OPA :8181, meshd :9451, all code |
| Mac2 | [ip] | Retired from TD | Old laptop, crashes under GPU load. Light tasks only. |
| Mac3 | [ip] | Storage + Creative | Media, exports, Serenity content |
| Mac4 | [ip] | Unity + Adobe | Unity 6 LTS installed (6000.0.72f1), Rosetta 2 installed, project at `[home-path]`, also has Adobe CC + bridge :9520 |
| Mac5 | [ip] | TouchDesigner + Compute | TD 2025.32460 installed, all TD files deployed, MLX :8100, fine-tune :9200, TB5 direct to Mac4 ([ip]↔[ip]) |

### Key Topology Change This Session
TouchDesigner moved from Mac2 → Mac5. Reasons: Mac2 crashes under load (laptop thermals), Mac5 has M4 GPU + desktop thermals + TB5 direct link to Mac4 (sub-ms latency for TD↔Unity streaming).

### What Was Installed This Session
- Mac4: Unity Hub (brew cask), Unity 6 LTS 6000.0.72f1 (arm64), Rosetta 2
- Mac5: Homebrew, TouchDesigner 2025.32460 (brew cask)

### Mesh Integration Points
| System | Connection | Port | Data |
|--------|-----------|------|------|
| bridged (Mocopi) | WebSocket | :9463 | motion.skeleton — 28 bones @ 30Hz |
| bridged (Audio) | WebSocket | :9463 | beat.* — BPM, beat position, energy |
| Echelon Hub | WebSocket | :9420 | predicted_grid, dynamics[0:104] |
| TD Bridge | HTTP | :9510 (Mac2 legacy) | Performance signals, skeleton relay |
| Unity → Mesh | OSC out | :9462 | Depth motion summary |
| multicam-server | WebSocket | :9404 (Mac1) | iPhone LiDAR depth frames |

---

6. FILE INVENTORY {#6-files}

DepthReactiveVisuals (`Desktop/DepthReactiveVisuals/`)

Assets/
  Scripts/
    Core/          (10 files) DepthSource, DepthProcessor, OpticalFlowProcessor,
                              FluidSimulator, IdleDetector, PoseTriggers,
                              CalibrationTool, DisplayManager, AttractMode,
                              PresetManager, FrameRecorder
    Audio/         (4 files)  AudioAnalyzer, AudioToFluid, AudioToPalette, AudioToVFX
    Visual/        (4 files)  ColorPaletteManager, ParticleBehaviorSwitcher,
                              SkeletonParticleEmitter, ChoreographyOverlay
    Network/       (5 files)  BridgedClient, BridgedAudioReceiver, MocopiReceiver,
                              EchelonReceiver, OSCSender
  Shaders/
    Compute/       (3 files)  DepthReprojection.compute, OpticalFlow.compute,
                              FluidSim.compute
TouchDesigner/     (2 files)  depth_reactive_builder.py, depth_fluid_sim.glsl
Docs/              (1 file)   CREATIVE-AGENCY-CONCEPT.md

LUME Evo3 Output (`Desktop/evo-cube-output/computational-choreography-device/`)

stage0-research.md          (185 lines)
stage1-path-a.md            (73 lines) — The Monolith
stage1-path-b.md            (63 lines) — The Bar
stage1-path-c.md            (65 lines) — The Portal
stage1-path-d.md            (63 lines) — The Mirror
stage1-path-e.md            (83 lines) — The Platform
stage1-path-f.md            (73 lines) — The Constellation
stage2-compound.md          (397 lines) — LUME synthesis
stage3-expand-master-plan.md (565 lines) — Risks, specs, roadmap

Creative Agency Site (`Desktop/creative-agency-site/`)

index.html                  (WebGL particles, warm obsidian palette)
package.json                (serve config)

TouchDesigner on Mac5 (`Desktop/cc-touchdesigner/`)

cc-echelon.toe              (main project)
depth_reactive_builder.py   (22-node network builder)
depth_fluid_sim.glsl        (Stable Fluids shader)
lidar_to_td.py              (iPhone LiDAR → TCP bridge)
lidar_network_builder.py    (multi-phone point cloud)
inject_live_network.py      (motion signals wiring)
cc_network_builder.py       (base network)
mocopi_to_td.py             (skeleton bridge)

### Unity Project on Mac4 (`[home-path]`)
All 24 .cs scripts + 3 .compute shaders deployed. Packages/manifest.json with URP + VFX Graph. ProjectSettings/ProjectVersion.txt points to 6000.0.72f1.

Reel Ingests (Obsidian Vault)

obsidian-vault/Inbox/2026-04-04/
  DWg5jJ0kq5J-instagram.md  (Hybrid Xperience — depth mirror, 2.4K likes)
  DWjfNVjkfbz-instagram.md  (Lazy Lighting — projection mapping app, 31K likes)
  DWmap5cEUD6-instagram.md   (stereo.drift — CV + body tracking + JS)
  DV9AVqmjLwE-instagram.md   (Augmenta — ISE Barcelona interactive room)

---

7. WHAT'S WORKING {#7-working}

• All 24 C# scripts compile (after asmdef removal and two fixes)
• All 3 compute shaders are mathematically complete (Stable Fluids, Horn-Schunck, depth reprojection)
• Unity 6 LTS installed on Mac4 with project deployed
• TouchDesigner installed on Mac5 with all files deployed
• LiDAR → TD bridge infrastructure already existed from prior work (lidar_to_td.py + lidar_network_builder.py)
• Mesh integration architecture is defined (bridged, Echelon, Mocopi, OSC)
• LUME product concept is fully evolved (Evo3, 9 files, 1,567 lines)
• 6 inspiration sources ingested and analyzed

---

8. WHAT'S BLOCKED / INCOMPLETE {#8-blocked}

### Unity (Mac4)
- [ ] No Unity scene created — need to create main scene, add GameObjects, wire SerializeField references
- [ ] No VFX Graph assets — need DepthParticles.vfx + behavior subgraphs (Wing, Jellyfish, Swirl)
- [ ] No URP Pipeline Asset — need to create and assign URP asset in Project Settings
- [ ] No Record3D integration — need to install Record3D Unity package, implement `Record3DDepthSource : DepthSource`
- [ ] No post-processing — need Bloom, Color Grading, Vignette, Chromatic Aberration on URP camera
- [ ] Not tested — scripts compile but nothing has been run yet

### TouchDesigner (Mac5)
- [ ] TD not opened yet — user hasn't opened cc-echelon.toe on Mac5
- [ ] depth_reactive_builder.py not executed — the 22-node network hasn't been built yet
- [ ] No iPhone LiDAR connected — Record3D not yet installed on iPhone
- [ ] Audio input not tested — mic/system audio → FFT → fluid reactivity not validated

### LUME Hardware
- [ ] No Jetson Orin Nano Super purchased — need to order ($249)
- [ ] No Orbbec Femto Bolt — user said ~2 weeks out
- [ ] No performance benchmark — critical R1 risk: can it run at 60fps?
- [ ] No iPad app — recommended as first validation step

### Agency
- [ ] Landing page needs redesign — Mohamed said too narrow, needs broader vision
- [ ] No Instagram page created
- [ ] No Vercel deployment
- [ ] No portfolio content captured yet — needs working system first

---

9. NEXT STEPS (Priority Order) {#9-next-steps}

### Immediate (today/tomorrow)
1. Open TD on Mac5, run `depth_reactive_builder.py`, validate fluid sim with noise input
2. Open Unity on Mac4, create main scene, wire up DepthProcessor + FluidSimulator + ColorPaletteManager, get something rendering
3. Install Record3D on iPhone ($3.99), test LiDAR streaming to Mac5 TD

### This Week
4. Create VFX Graph assets in Unity (DepthParticles.vfx with attribute maps from fluid velocity)
5. Configure URP pipeline + post-processing (Bloom, Color Grading)
6. Test iPhone LiDAR → TD → fluid sim → particles end-to-end on Mac5
7. Order Jetson Orin Nano Super Developer Kit ($249) for LUME R1 benchmark

### This Month
8. Port visual pipeline to run on Jetson (CUDA/Vulkan if Unity doesn't hit 60fps)
9. Build iPad app prototype (Metal + LiDAR, validates demand)
10. Capture portfolio content for agency Instagram
11. Redesign agency landing page with broader vision, deploy to Vercel
12. Prepare June pop-up as Kickstarter demo

### Fundraising Path
13. Kickstarter campaign (Month 4, target $300K+)
14. Pre-seed raise (Month 5, $500K at $10M cap)
15. First 500 units manufactured (Month 9)

---

10. REFERENCE MATERIAL {#10-references}

### Key External References
- keijiro/StableFluids — GPU fluid sim for Unity (reference implementation)
- keijiro/Rsvfx — Depth → VFX Graph attribute maps
- marek-simonik/record3d_unity_demo — iPhone RGBD streaming to Unity
- orbbec/OrbbecSDK_v2 — C API for Femto Bolt
- Augmenta Unity Plugin — Asset Store, OSC tracking data input
- Augmenta TD Plugin — GitHub, real-time tracking data

### Market Data
- US dance studios: ~14,600
- US museums/galleries: ~8,000-11,000
- US event production companies: ~3,400+
- US nightclubs/venues: ~60,000
- Immersive art sector: $1B+ annually
- Mercer Labs: $38-50/visit, 36K sqft, 15 rooms, NYC
- teamLab: 2.3M visitors/year

### Existing Evo3 Master Plan
The original depth-reactive visuals plan is at `[home-path]` — 6 waves, detailed task lists. This handoff supersedes it with the LUME product direction.

### Memory Updated
- `mesh-architecture-v2.md` — updated with Mac5=TD, Mac2=retired, Mac4=Unity+Adobe
- `project_creative-agency.md` — new memory for agency concept
- `CREATIVE-AGENCY-CONCEPT.md` — full doc with all inspiration references

---

End of handoff. Total artifacts: 30 code files, 9 evolution files, 1 landing page, 1 agency concept doc, 4 ingested reels, 1 handoff doc. Three interconnected initiatives sharing one codebase.