LumeBodyTruth Contract

LumeBodyTruth Contract

v0.1 | 2026-05-24 | Architecture contract for DYK visuals + K11 DJ control

Implementation status: v0 read endpoint is live in `multicam-server` as
`GET /body-truth`.

`LumeBodyTruth` is the perception authority above the camera/sensor feeds. It
answers one question for every consumer: "what do we trust about the performer
right now?"

It is not a custom neural network yet. It is a fused state layer that combines
depth, RGB, mocopi, iPhone/SAN, and MotionMix signals into one stable body
truth object.

1. Current Machine Roles

2. Existing Pieces To Build On

MotionMix already emits `fused_body_state` from `multicam-server/src/main.rs`.
That state includes useful fields such as:

• `source`
• `source_authority`
• `joint_count`
• `camera_count`
• `joint_coverage`
• `triangulation_strength`
• `avg_energy`
• `avg_motion`
• `avg_framing`
• `avg_coverage`
• `stillness_score`
• `semantic`
• `joints_3d`

`LumeBodyTruth` should wrap and harden this. The goal is not to discard
`fused_body_state`; the goal is to make a stricter performer-truth contract
that DYK and K11 can both trust.

3. Source Inputs

flowchart TD
  MegaDepth["Mac4 Mega depth body/matte"] --> Truth["LumeBodyTruth"]
  MegaRGB["Mac4 Mega RGB pose/person evidence"] --> Truth
  Arducam["Arducam UVC/RGB or stereo/depth person detector"] --> Truth
  Insta360["Insta360 wide-angle RGB/body evidence"] --> Truth
  Mac2Insta["Mac2 Insta360 room-wide angle evidence"] --> Truth
  Mocopi["Sony mocopi 27-bone skeleton"] --> Truth
  K11Pose["K11 webcam/Arducam UVC/Bolt RGB pose assist"] --> Truth
  IPhone["iPhone MotionMix/SAN"] --> Truth
  Truth --> DYK["Mac4 Unity DYK visuals"]
  Truth --> K11["K11 Rekordbox/AirDeck bridge"]
  Truth --> Record["motion truth recording"]

Input policy:

- Depth/matte is best for the beautiful real body and coarse 3D occupancy.
- RGB/person segmentation is best as a human-presence veto.
- Mocopi is best for stable body skeleton, torso, head, hips, and limbs.
- K11 camera is best for DJ-area hand intent and AirDeck zones.
- iPhone/SAN is best as supplemental motion and training evidence.
- Insta360 is best as wide-angle RGB/body evidence and session recording. It is
useful for multi-view BodyTruth, but it does not replace depth unless another
depth/stereo source is present.
- Mac2-hosted Insta360 is best as a room-wide reconstruction view when the
camera is physically plugged into Mac2. It is evidence-only and must be
labeled as `mac2_insta360_room_wide`.

Sensor placement policy:

- K11 does not require a Femto/Bolt class depth camera for Pose Coach. A normal
UVC webcam, UVC Arducam, or low-latency UVC/HDMI Insta360 feed is acceptable
when the frame clearly sees the full body and the goal is command-safe 2D
pose/intent gating.
- Put the Femto Bolt, Femto Mega, or any Arducam that depends on stereo/depth
SDK processing on Mac4 when the goal is richer BodyTruth: multi-angle body
shape, depth confidence, occlusion recovery, visuals, and MRT2-style audio
mapping.
- K11 should receive only the distilled safety signal it needs for
AirDeck/Rekordbox decisions. Mac4 may observe, fuse, render, and map motion,
but it must remain read-only and must not become a command authority.
- A good default is one simple K11 camera aimed at the DJ/control area plus two
Mac4 cameras at different body angles. The Mac4 pair can be Bolt/Mega,
Arducam plus Mega, Insta360 plus depth, or any equivalent RGB/depth mix.
- When the Insta360 is connected to Mac2, Mac2 becomes the room-wide evidence
host. That lane may feed calibration, review, and later Mac5 reconstruction,
but it must not send Rekordbox commands and should not be accepted by the K11
live source filter.

4. Output Schema

Minimum `LumeBodyTruth` JSON:

{
  "type": "lume_body_truth",
  "schema": "lume.body_truth.v0",
  "timestamp": 1779639382527,
  "present": true,
  "confidence": 0.86,
  "authority": "mocopi_rgb_depth",
  "mode": "active",
  "tracking_lost": false,
  "center": { "x": 0.12, "y": 0.48, "z": 1.72 },
  "bounds": { "x": 0.9, "y": 1.55, "z": 0.45 },
  "velocity": 0.18,
  "stillness": 0.72,
  "burst": 0.08,
  "posture": "upright",
  "seated": false,
  "gesture_intent": {
    "name": "none",
    "confidence": 0.0,
    "deck": null,
    "command": null
  },
  "sources": {
    "depth": { "present": true, "confidence": 0.82, "fresh_ms": 24 },
    "rgb": { "present": true, "confidence": 0.91, "fresh_ms": 31 },
    "mocopi": { "present": false, "confidence": 0.0, "fresh_ms": null },
    "k11_pose": { "present": true, "confidence": 0.76, "fresh_ms": 22 },
    "iphone": { "present": false, "confidence": 0.0, "fresh_ms": null }
  },
  "raw_ref": {
    "fused_body_state": true,
    "motion_record_id": null
  }
}

5. State Machine

`mode` is the stable state every consumer should branch on:

6. Confidence Rules

Initial v0 rules should be boring and explicit:

- `present=true` requires at least one fresh human source with confidence >= 0.55.
- `confidence >= 0.75` requires either:
- mocopi fresh with enough canonical joints, or
- depth present plus RGB person verifier, or
- two independent camera/person sources.
- `tracking_lost=true` when the previous confident body is stale but the timeout
window has not fully expired.
- `burst` should require both human presence and motion energy; raw depth shimmer
alone cannot create burst.
- K11 commands require `present=true`, `tracking_lost=false`, and
`gesture_intent.confidence >= command threshold`.

7. Consumer Contracts

Mac4 Unity / DYK

DYK should use `LumeBodyTruth` for gating and intensity:

• `present` gates whether body-reactive effects are allowed.
• `confidence` scales visual intensity.
• `stillness` controls freeze/crystallization behaviors.
• `burst` drives particles, trails, BurstWig, and wall accents.
• `tracking_lost` fades or holds last safe visual state.

DYK should not directly infer performer truth from scattered globals once this
contract is live.

Current Mac4-first bridge:

python3 Desktop/MotionMix/multicam-server/mac4_bodytruth_output_bridge.py \
  --bodytruth-url http://[ip]:9405/body-truth \
  --enthea-map <bundle>/derived/enthea/bodytruth_control_map.json \
  --mrt2-map <bundle>/derived/mrt2/control_map.json \
  --state-out Desktop/MotionMix/td-bridge/mac4_lume_output_state.json

The bridge writes `lume.mac4_first.live_output_state.v1` for ENTHEA/Unity and
MRT2-style consumers. It is a read-only BodyTruth consumer: no keyboard events,
no Rekordbox commands, no AirDeck promotion, and no BodyTruth writes.

Every first-capture bundle now carries the intended camera/device assignment as
`derived/mac4_first/device_topology.json`. That sidecar makes the architecture
auditable before Mac5 reconstruction: K11 remains the local Pose Coach and
Rekordbox/AirDeck command gate, Mac4 receives richer BodyTruth camera inputs
for ENTHEA/Unity/MRT2/DEMON output lanes, and Mac5 stays offline until the
Mac4-first live gate is green.

Mac2 now has a matching angle-capture lane for the physically connected
Insta360. Its session bundles write
`derived/reconstruction/device_topology.json` with
`source_id=mac2_insta360_room_wide`,
`command_authority_changed=false`, and `k11_only_command_gate=true`. Mac2 may
therefore add a wider view for occlusion recovery, camera-angle diversity, and
future reconstruction, but it remains outside the live Rekordbox command path.

Mac4 live runtime checkpoint:

BodyTruth service: http://[ip]:9405/body-truth
Output state API:  http://[ip]:9415/state
OBS visual URL:    http://[ip]:9415/visual
MRT2 Web MIDI:     http://[ip]:9415/mrt2 on Mac4
MRT2 inspection:   http://[ip]:9415/mrt2
MRT2 proof target: LUME MRT2 Proof Sink (Mac4 CoreMIDI virtual destination)
MRT2 browser proof endpoint: http://[ip]:9415/mrt2-proof
MRT2 proof status: [home-path]
MRT2 proof log:    [home-path]
MRT2 browser proof: [home-path]
Mac4 state file:   [home-path]
Mac4 JSONL log:    [home-path]
Local readiness:   [home-path]
LaunchAgents:      com.lume.bodytruth-control, com.lume.mac4-output-bridge, com.lume.mrt2-proof-sink

The HTTP state API sends `Access-Control-Allow-Origin: *` so ENTHEA, OBS browser
sources, Unity editor tooling, and small local dashboards can poll it directly.
The `/mrt2` page must be opened locally on Mac4 for Web MIDI permission
(`http://[ip]:9415/mrt2`). It sends only MIDI CC values from `mrt2.cc`
after a browser grants MIDI access and a non-Rekordbox output is selected.
Mac4 now also keeps a receive-only CoreMIDI target named `LUME MRT2 Proof Sink`
alive through `com.lume.mrt2-proof-sink`. It is the default safe proof output
for Chrome/Web MIDI selection when no DAW/MRT2 target is attached. It logs
received CC events, sends no notes, sends no keys, and cannot send Rekordbox
commands.

The `/mrt2` page now posts a browser proof to `/mrt2-proof` after it has Web
MIDI access, a selected output, the K11 safety lock, and at least one sent CC.
Open this exact URL on Mac4 when proving the browser path:

http://[ip]:9415/mrt2?output=LUME%20MRT2%20Proof%20Sink

The bridge writes `mrt2_browser_proof.json` with
`proof_kind=browser_web_midi_runtime` only for the real browser path. Synthetic
endpoint checks are marked `proof_kind=endpoint_smoke`, so they cannot be
confused with a human-granted Chrome Web MIDI permission proof.
The Mac4-first goal audit now reads that proof file over SSH during full live
audits and fails unless it is the real `browser_web_midi_runtime` proof for the
receive-only `LUME MRT2 Proof Sink`.

After Mac4 is physically back and SSH is reachable, Mac1 can sync the staged
Mac4-first service payload and run the local installer:

[home]/Desktop/MotionMix/multicam-server/bootstrap_mac4_first_local_services.sh --host mac4

If the staged Mac4-first files are already present on Mac4, the same local
read-only services can be installed directly on Mac4 before the Mac1 completion
gate:

bash [home-path] --run-probe

The bootstrap and installer stage/start the BodyTruth control plane, output
bridge, receive-only CoreMIDI proof sink, and local readiness probe. The
installer writes
`[home-path]` and preserves
`mac4_sends_rekordbox_commands=false`.

After any browser proof attempt, the local probe can be rerun directly:

python3 [home-path]

The probe report is read-only. It validates local launchd services, BodyTruth
`:9405`, output bridge `:9415`, `/visual`, `/mrt2`, the proof sink status, and
the browser Web MIDI proof file while preserving
`mac4_sends_rekordbox_commands=false`.

For Codex-driven Chrome proof attempts on Mac4, prefer the grant-only CDP mode:

LUME_PERMISSION_MODE=grant LUME_CHROME_DEBUG_PORT=9224 \
  node [home-path]

From Mac1, the repeatable live wrapper is:

[home]/Desktop/MotionMix/multicam-server/run_mac4_mrt2_live_browser_proof.sh

It syncs only `run_mac4_mrt2_browser_runtime_proof.js` to Mac4, starts an
isolated Mac4 Chrome DevTools session when needed, targets
`http://[ip]:9415`, validates `proof_kind=browser_web_midi_runtime`, and
re-checks that K11 remains the only Rekordbox command authority.

Before Mac5 heavy reconstruction, run the completion gate:

python3 [home]/Desktop/lume-commerce/viz/lume-pcloud/tools/lume-mac5-reconstruction/run_lume_mac4_first_completion_gate.py \
  --output /tmp/lume_mac4_first_completion_gate_latest.json

Mac5 heavy reconstruction stays gated until that report says
`completion_ready=true` and `mac5_heavy_reconstruction_allowed=true`.
The gate first checks Mac4 reachability on SSH `:22`, BodyTruth `:9405`, and
the output bridge `:9415`. If those ports are unavailable, it skips the wrapper
and full audit and writes a fast failure report instead of pretending the live
proof was attempted.
For a cheap reachability-only pass:

python3 [home]/Desktop/lume-commerce/viz/lume-pcloud/tools/lume-mac5-reconstruction/run_lume_mac4_first_completion_gate.py \
  --preflight-only \
  --output /tmp/lume_mac4_first_preflight_latest.json

For a one-shot watch pass that only runs the full gate after preflight is ready:

python3 [home]/Desktop/lume-commerce/viz/lume-pcloud/tools/lume-mac5-reconstruction/watch_lume_mac4_first_completion_gate.py \
  --max-attempts 1 \
  --output /tmp/lume_mac4_first_completion_gate_latest.json \
  --watch-log /tmp/lume_mac4_first_completion_gate_watch.jsonl

`LUME_PERMISSION_MODE=set` remains available for Chrome builds where
`Browser.setPermission` behaves correctly, but the grant mode avoids the page
runtime freeze seen with the current Mac4 Chrome permission override path.

Unity ingestion is now represented by
`Assets/Scripts/LumeBodyTruthStateClient.cs` in `lume-pcloud`. It polls
`http://[ip]:9415/state`, requires the K11 safety lock, and publishes
smoothed BodyTruth controls to LUME shader globals for visual passes. It does
not send commands.

K11 Rekordbox / AirDeck

K11 should continue to own the final command gate:

• AirDeck/voice/keyboard dispatch stays inside `lume_rekordbox_bridge.py`.
• `LumeBodyTruth` can enable, disable, and bias gesture recognition.
• Raw mocopi or Unity should not send Rekordbox commands directly.
• Self-play remains the rehearsal lane for gesture chains before promotion.

8. Rollout Plan

1. Done: add `/body-truth` endpoint to MotionMix as a wrapper over current
`/skeleton-3d` and pose signal data.
2. Done: keep v0 read-only: no commands, no DJ side effects.
3. Done: add Mac4 read-only output bridge for `/body-truth` to ENTHEA/Unity,
OBS browser visuals, and MRT2-style Web MIDI CC state.
4. Done: add Mac4 receive-only CoreMIDI proof sink for MRT2/Web MIDI CC output.
5. Done: add a machine-verifiable goal audit:
`tools/lume-mac5-reconstruction/audit_lume_mac4_first_goal.py`.
6. Done: add bundle-level `derived/mac4_first/device_topology.json` so camera
ownership and the K11-only command gate are checked by the output-artifact
verifier.
7. Point K11 bridge at `/body-truth` as an extra safety input while keeping
local AirDeck gesture detection.
8. Finish Sony/mocopi proof so `/skeleton-3d` becomes fresh `source=mocopi`
when mocopi is active.
9. Record `LumeBodyTruth` frames to SQLite/JSONL alongside existing motion
records.
10. Train custom gesture/motion models only after the v0 truth stream is stable
enough to produce trustworthy labels.

9. Non-Goals For v0

- Do not train a custom neural network.
- Do not let Unity directly control Rekordbox.
- Do not remove existing K11 AirDeck safety checks.
- Do not require full-body visibility; seated and torso-only states must be
valid if confidence is high enough.
- Do not make mocopi mandatory; it is one strong source, not the only source.

10. Success Criteria

v0 is working when:

• `/body-truth` returns a stable object even when no body is present.
• A still performer does not produce false burst events.
• Leaving frame moves to `tracking_lost`, then `absent`, without command spam.
• Seated/torso-only posture can still produce `present=true`.
• DYK and K11 both consume the same truth signal.
• Rekordbox commands still only pass through the K11 bridge.