Codex context — where LUME started + how it got here

Codex context — where LUME started + how it got here

> Topology correction, 2026-04-26: Mac4 is the current Unity Editor / GUI smoke-test host and real-Femto capture host. Mac5 is still the synthpub LaunchAgent / synthetic fallback host. Do not blindly replace all Mac5 references; see `software/demo/TOPOLOGY-CORRECTION-2026-04-26.md`.

_Read this BEFORE `CODEX-PLAN.md`. ARCHITECTURE.md describes WHAT is
built; this file describes WHY it's built that way._

You're inheriting a 22-commit pipeline. The destination matters less
than the load-bearing decisions made along the way — a dozen of them
look "obvious" only because they were extracted painfully from
constraints you don't see in the final tree. This file is the
decision log.

Origin — 2026-04-25 Saturday demo deadline

LUME v1's scope was anchored on a Saturday-night ship target. The
work that became Wave 1 was sized to fit that window:

- 1-3 vertical monitors, depth camera (Femto Mega) + audio (UMA-8),
Unity URP scene rendering an audio-reactive point cloud.
- Form factor: a 500mm wide × 120mm tall × 85mm deep bar (Sonos-style).
- Compute target initially: NVIDIA Jetson AGX Orin 64GB ($1999) inside
the bar.

The 5-step `chain:genesis` skill was used to scope the build:
explore → evo³ → creative-forge → divergent-rail → meta-review.
Output: `lume-v1-rail-plan.md` with 5 waves × 17 tracks. Verdict
SHIP-GO Sat 8pm. P0 issues: NONE. P1×3 (audio pointer, mic
permission, caffeinate). P2×5 deferred to Wave 2-4.

That demo's outcome is not yet captured in memory — when you
catch the user in person, ask them how it landed.

Hardware journey

Jetson AGX Orin → GMKtec NucBox K11

The Jetson Orin was dropped before Sat-demo, mid-Wave-1. Reason:
Unity ARM64 Linux is technically "embedded platform" (paid plan,
not promised in standard licenses) and depending on a license
classification we don't fully control was an unacceptable risk.

K11 chosen instead — Ryzen 9 8945HS + Radeon 780M + 32GB DDR5,
$899.99. Linux x86_64 = standard Unity Player target, license risk
goes to zero. Saves $1099/unit vs Orin. Same physical envelope
(roughly).

Implication for code: the LUME pipeline runs on Linux x86_64,
NOT ARM64. Any Wave 5+ port that assumes Jetson SDK / CUDA needs
to be rewritten for AMD/Radeon. Compute shaders compile on Vulkan;
no CUDA dependency in the codebase.

Femto Mega → Femto Bolt (deferred to Wave 5)

Both Orbbec depth cameras share the same Sony IMX556 ToF sensor.
Mega has an internal Jetson Nano + UMA-7 mic array; Bolt
is USB-C only, no internal compute, no mics. With the K11 pivot,
the Mega's onboard Jetson Nano becomes dead weight (K11 does all
processing), and the LUMF audio-sidecar architecture (Wave 4-B)
makes the Mega's mic array redundant.

Bolt wins on every axis with K11 hosting: ~$700 cheaper, 35-45mm
thinner enclosure possible, simpler USB-C cabling. The reason it's
deferred: the Mega → :9710 capture pipeline already works for the
Sat-demo floor; swapping cameras pre-demo was unjustified risk.
Wave 5 is when the K11 lands and the bar gets re-cut anyway, so
that's the natural moment for the Bolt swap.

Wave evolution

Each wave was scoped against what the Sat-demo floor could safely
land on top of. The invariant: Wave 1 cloud-mode path stays
bit-preserved through every subsequent wave. Disabling Wave 2/3/4/5
components on a GameObject reverts to Wave 1 with no code changes.
This isn't aesthetic — it's the demo-rescue path.

Wave 1 — audio-reactive cloud (Sat-demo floor)

`pointcloud_pub.py` emits LUME-magic 16-byte-header datagrams of
CPU-projected XYZ+RGBA points to UDP :9700. `LumeUdpReceiver` reassembles
per-frame, `LumePointRenderer` URP-instances them, `LumeAudioReactor`
captures mic + drives `_AudioLevels`/`_OutlineFlash`/`_GlobalHueCycle`
shader globals. Cyan-pink palette, hue cycle 30s, beat-flash on RMS
above threshold.

Anchor commits: `2f63709e`, `751fcaf3` (two-channel reactivity),
`e791279b` (LumeVfxEditor scaffolding).

Wave 2 — GPU depth reprojection + LUMD wire format

CPU-projection in `pointcloud_pub.py` was eating an entire core. We
moved depth reprojection to a Unity compute kernel (`LumeDepthReproject.compute`,
8×8 threadgroup, ByteAddressBuffer u16 unpack, pinhole back-projection
into world space).

The publisher now emits raw uint16 depth + intrinsics over a NEW
LUMD magic (`0x4C554D44`) at full Femto resolution (1024×1024) with
default 716-pixel chunks (= MTU-clean 1472 bytes, no IP fragmentation,
clean macOS sendto).

Decision: every LUMD datagram carries intrinsics in its header.
A dropped packet can never blind a frame — receiver always knows how
to project the pixels it did get. The ~36-byte overhead is dwarfed by
the latency benefit of stateless decoders.

Anchor commit: `7387abb2`.

Wave 3 — frame-diff motion → SetInnerSpread

Duncan's "two-channel reactivity" model: outline = audio (form),
inner = motion (action). Wave 3 wires the inner channel.

Decision: frame-diff scalar over Lucas-Kanade pyramid initially.
The shader at this point read `_InnerSpread` as a single float uniform.
Reducing LK's RG16F output to a scalar = mean(|d_curr - d_prev|) in
the limit. Same answer, ~10× less compute. We shipped frame-diff;
LK pyramid landed in Wave 4 when the shader started consuming a
flow texture.

Anchor commit: `b8a750d7`.

Wave 4 — 10 sub-tracks (A through J)

Wave 4 was supposed to be one critical track (LK pyramid). It became
ten. Reason: while one pulse-runner subagent shipped the LK kernel
(track A), parallel work demanded LUMF audio sidecar (track B),
Unity-side LUMF receiver (track C), VFX Graph bootstrap (track D2 —
D1 stalled), inspector tests (track E), Mac5 health-check (track F),
pytest @slow marker (track G), VFX runtime bridge (track H —
closing a gap track D2 left), golden-byte regression (track I),
inspector e2e (track J).

Two architectural calls landed in this wave:

1. LUMF audio sidecar runs in Python, not Unity. Unity's
`AudioSource.GetSpectrumData` has ~40ms game-thread latency on Mac.
Python+numpy FFT clocks ~6-10ms. We commit to the latency win and
ship a separate `:9701` UDP stream + a Unity-side `LumeAudioFftReceiver`
that overrides `_AudioLevels`/`_OutlineFlash` when fresh, yields
silently to `LumeAudioReactor`'s mic path when stale.

2. Components coexist via execution order, not coupling.
`LumeAudioFftReceiver` runs at `[DefaultExecutionOrder(200)]`
strictly after `LumeAudioReactor` (default 0), so it overwrites
the same shader globals the reactor just published. No `if(use_sidecar)`
branching anywhere; failure mode = stop overwriting, the reactor
takes back over the next frame.

Anchor commits: `ca975000` (P1), `5d4e61c6` (P2), `d55d025a` (P3),
`976e594a` (B), `e36956f7` (C), `6d312a95` (D2), `debca656` (E),
`aa2f5c40` (F), `e0ac2fbb` (G), `cf9becc4` (H), `27b15b9f` (I),
`ef7e42da` (J).

Wave 5 — Duncan-stack closure (3 of 5 done)

Per `DUNCAN-GAP-ANALYSIS.md`, Wave 5's priority order was: impulse
channel, calibration UI, fluid sim, SuperHot motion-gating, marching
cubes. We shipped 1, 2, 4 (motion-gating before fluid sim because
it's prerequisite to fluid sim's `_LumeTimeScale` consumer).

Decision: motion-gate publishes a private `_LumeTimeScale` global
instead of touching `Time.timeScale`. Touching `Time.timeScale`
slows audio playback. Duncan's installations don't slow the music.
The fluid sim opts in by multiplying its delta-time against the
global; everything else (audio, UI, particles outside the gate) runs
at real-time.

Decision: calibration UI uses runtime reflection on `[SerializeField]`
private fields. Every component would otherwise need 16+ public
properties — a change-the-world commit. Reflection gets us per-install
tweak without a single line of edit on the existing components.
Auto-saves to `Application.persistentDataPath/lume-calibration.json`.

Anchor commits: `057b87d2` (impulse), `32b02358` (calibration),
`a6e7fc16` (motion-gate), `e2089c56` (ARCHITECTURE.md consolidation).

Decision log — short list of "why this looks weird"

Don't-touch list (load-bearing constraints)

- `pointcloud_pub.py` — Mac5 LaunchAgent runtime depends on its CLI.
- `LumeUdpReceiver.cs` — magic-byte dispatch is settled.
- `LumeAudioReactor.cs` — parallel session contract; use the public
setters (`SetInnerSpread`); don't modify the class.
- Wave 1 cloud path — must stay bit-preserved.
- Mac4 — TCC-blocked (no Camera permission), no SSH-with-camera-access
work possible. Stay on `--synthetic` / `--synthetic-depth`.
- `[home-path]` — clear if no live `git` process. Safe to rm.
- No git push. Mac1 main local commits only.

Meta — how to use this doc

When you hit a decision that seems arbitrary, search this doc for the
keyword first. If the decision isn't here, write a one-line entry in
the Decision Log table at the bottom and move on. The doc grows; it
doesn't shrink.

When you make a NEW decision Codex-side, append to the table — same
rules. Future-you (or Claude / a third agent) reading this will save
hours of head-scratching.

What you actually need to do

→ Read `CODEX-PLAN.md` next.