LUME Rehearsal Capture Overwhelming Update

LUME Rehearsal Capture Overwhelming Update

Date: 2026-05-28
Primary live host: Mac4
Primary storage target: K11
Reference project root on Mac4: `[home]/dev/lume-commerce-live/viz/lume-pcloud`
Document destination on Mac1: `[home]/Desktop/lume-commerce/viz/lume-pcloud/Docs/LUME_REHEARSAL_CAPTURE_OVERWHELMING_UPDATE_2026-05-28.md`

Conversion companion:

Docs/LUME_K11_MAC4_MAC5_POSE_COACH_CAPTURE_CONVERGENCE_2026-05-28.md

The companion document converts this Tier 0 capture lane into the shared
K11/Mac4/Mac5 rehearsal-bundle architecture and explains how K11 Pose Coach
fits into the same storage/training path.

Executive State

The rehearsal-capture stack has crossed from concept into a working Tier 0 capture lane. The critical achievement is not that every sensor is perfect yet. The critical achievement is that LUME now has a boring, survivable, inspectable recording path that can record a long physical rehearsal, preserve labels, survive interruption, write explicit session metadata, and route the completed bundle to K11 with remote checksum verification. This matters because every more advanced choreography layer depends on clean session evidence. The machine can now collect a body session without requiring Unity, MotionMix, mocopi, Web Lab, iPhones, or any DJ command path to be alive.

The safe current rehearsal route is:

Mac4 Insta360 X4 camera
  -> tmux-launched Tier 0 recorder
  -> local session folder under docs/dance-sessions
  -> verified archive/checksum/manifest upload to K11
  -> later analysis on Mac4/Mac5/K11

The reason `tmux` is explicit in the route is important. Direct Codex-launched ffmpeg could list the physical cameras but got zero frames from them. The same ffmpeg capture launched inside `tmux` successfully captured an Insta360 frame and then full MP4 test sessions. macOS Camera privacy already grants access to `Terminal` and `tmux`, so the production route intentionally uses `tmux` as the camera-permission context. This is not a hack around recording quality. It is an operating-system permission boundary being made explicit and stable.

K11 is now a storage target, not a live dependency. The current stack does not live-write the only copy directly over the network, because that would make the rehearsal vulnerable to Wi-Fi, Tailscale, SSH, sleep, and transient Windows file-system stalls. The chosen route records locally first, then immediately verifies K11 upload after the take. This is the correct reliability posture for a weighted freestyle session: capture first, transport second, analyze third.

Machine Roles

Mac4 remains the physical sensor and live visual machine. It owns the Femto/Mega body feed, Unity pCloud/DYK visuals, Sony mocopi input when charged, Web Fluid Lab experiments, and the local camera capture controller. For the rehearsal capture lane, Mac4 owns the camera device and produces the local session folder. Mac4 should not be treated as infinite storage. It currently has roughly 30 GB free on `/System/Volumes/Data`, which is enough for tests but tight for a long session. The full weighted take should ideally start with at least 80 to 100 GB free.

K11 is now the central dance-session storage target. Its role is to receive completed session bundles, retain incoming archives, checksums, manifests, processed outputs, exports, and logs. K11 must remain separate from Rekordbox control. The rehearsal session upload path does not restart or modify `MotionMixLumeBridge`, does not touch `C:\temp\lume_rekordbox_bridge.py`, and does not send MIDI, keyboard, or DJ commands. K11 can later run indexing, manifest validation, and storage management, but the rehearsal recorder sees it only as storage.

Mac1 is the documentation and coordination node for this update. This document is being saved into the Mac1 copy of the pCloud project because Mac1 appears to have a mirrored or older copy of `[home]/Desktop/lume-commerce/viz/lume-pcloud`. It should become the durable human/agent handoff for what was actually proven.

Mac5 remains the offline reconstruction worker target. SAM3DBody-cpp belongs there as an after-the-fact processor for uploaded rehearsal videos. It should not be pulled into the live rehearsal loop. Its best role is overnight or background analysis: sample frames, reconstruct body features, emit CSV/BVH/JSONL artifacts, then feed templates back to Unity/Web Lab.

MotionMix remains optional Tier 3 telemetry. It can enrich sessions with iPhone/body-truth data when it is up, but the recording must not depend on it. The MotionMix sidecar was tested with a bad hub URL and behaved correctly: it wrote an unavailable telemetry row and did not invalidate the local session. This is the right failure mode. MotionMix can be useful evidence, not a single point of failure.

What Was Built

The Tier 0 recorder lives at:

[home]/dev/lume-commerce-live/viz/lume-pcloud/tools/lume-dance-capture/lume_dance_recorder.py

It captures a single AVFoundation camera through ffmpeg into one continuous MP4. It writes marker labels to JSONL. It writes `session.json`, `recorder.log`, `ffmpeg.log`, and ffmpeg progress logs. It can inspect sessions after the fact. It estimates disk usage when a duration is known. It warns when free space is below 50 GB and refuses below 20 GB unless `--force` is used. It uses a recorder lock to prevent overlapping recorder runs. It finalizes interrupted sessions as explicit `incomplete` sessions rather than silently leaving ambiguous folders behind.

The tmux camera launcher lives at:

[home]/dev/lume-commerce-live/viz/lume-pcloud/tools/lume-dance-capture/run_dance_recorder_tmux.sh

This launcher exists because physical camera capture works from the `tmux` permission context. It starts the recorder inside a named tmux session and logs to `/tmp/lume-dance-recorder-tmux.log` by default. It is the preferred path for Mac4 physical-camera takes.

The K11 routed rehearsal launcher lives at:

[home]/dev/lume-commerce-live/viz/lume-pcloud/tools/lume-dance-capture/run_k11_weighted_take_tmux.sh

This is the high-level command for the real take. It wraps the tmux launcher, selects the default Insta360 X4 camera, uses 1920x1080, expects a long session, and enables K11 upload. It can be overridden with environment variables such as `LUME_CAMERA_NAME`, `LUME_VIDEO_SIZE`, `LUME_SESSION_NAME`, `LUME_EXPECTED_DURATION`, and `K11_SSH_TARGET`.

The K11 transport script lives at:

[home]/dev/lume-commerce-live/viz/lume-pcloud/tools/lume-data-transport/push-session-to-k11.sh

It archives a session folder, computes SHA-256, writes a transport manifest, creates K11 incoming/manifest directories, transfers archive/checksum/manifest, verifies the remote archive hash, and updates local `session.json` with the K11 result. Its default SSH target is:

mohamed diomande@[ip]

The short SSH alias `k11` was not available from the Mac4 shell during the verification pass, so the explicit Windows account plus Tailscale IP is the proven target.

Session Folder Contract

The current Tier 0 session folder format is:

docs/dance-sessions/YYYYMMDD-HHMMSS-session-name/
  session.json
  raw/
    video/
      webcam_main.mp4
    labels/
      labels.jsonl
    motionmix/
      motionmix_state.jsonl        optional
      devices.json                 optional
  logs/
    recorder.log
    ffmpeg.log
    ffmpeg-progress.log
    motionmix-capture.log          optional

The session format is intentionally boring. This is a strength. It means a failed take still has enough state to diagnose what happened. A complete take has a normal MP4 plus labels. An incomplete take is not thrown away. It is marked explicitly and can still contain useful partial video, labels, and logs.

`session.json` is the spine of the session. It records schema, session ID, session name, host, project root, camera backend, camera index/name, paths, status, incomplete flag, label count, video probe information, disk preflight, K11 push result, and final reason. The important policy is that `status` must be explicit. There should be no mystery folder where an agent has to guess whether the take succeeded.

`labels.jsonl` is the live marker stream. Marker rows include `type`, `timestamp_unix`, `elapsed_sec`, `key`, `label`, `session_id`, and `source`. Required labels are:

1 = keep_signature
2 = wave_color
3 = burst_high_energy
4 = torso_lean_weight_shift
5 = weighted_slow_power_hold
0 = still_recovery_baseline
9 = bad_false_reactive

These labels are not clips. The session is one long take. Labels are time anchors for later window extraction. The intended analysis window is built around the labels after the session, not during capture.

Verified Evidence

The first successful physical-camera proof was a direct one-frame ffmpeg capture launched under `tmux`. It produced:

/tmp/lume-tmux-camera-test.jpg

The image was 1920x1080 from the Insta360 X4. The ffmpeg log showed a real AVFoundation input:

Input #0, avfoundation, from 'Insta360 X4:none'
Video: rawvideo, nv12, 1920x1080

This proved that the camera, USB layer, AVFoundation device, and ffmpeg stack could work. It also proved that the failure was not the camera being unplugged or absent.

The first recorder-level physical-camera tmux test completed here:

[home]/dev/lume-commerce-live/viz/lume-pcloud/docs/dance-sessions/20260527-192225-camera-unblock-test-insta360-tmux

It produced a complete session, three labels, and an H.264 1920x1080 MP4. This proved that the actual Tier 0 recorder could capture physical camera video when launched in the correct permission context.

The wrapper-level test completed here:

[home]/dev/lume-commerce-live/viz/lume-pcloud/docs/dance-sessions/20260527-192335-camera-wrapper-test-insta360

It produced:

status: complete
labels: 3
video_bytes: 7996395
duration: 7.693367
codec: h264
frame: 1920x1080

This proved that `run_dance_recorder_tmux.sh` is viable as an operator path.

The first K11 routed smoke completed here:

[home]/dev/lume-commerce-live/viz/lume-pcloud/docs/dance-sessions/20260527-194938-k11-route-smoke-insta360

It recorded from the Insta360, completed locally, and pushed to K11. The second K11 routed smoke, after metadata polish, completed here:

[home]/dev/lume-commerce-live/viz/lume-pcloud/docs/dance-sessions/20260527-195131-k11-route-smoke2-insta360

It produced:

status: complete
labels: 2
video_bytes: 4118089
duration: 4.838067
pushed_to_k11: true
push_ok: true
remote archive: C:/lume/dance-sessions/_incoming/k11-route-smoke2-insta360.tar.gz
remote manifest: C:/lume/dance-sessions/_manifests/k11-route-smoke2-insta360.transport.json
sha256: 2eeb5efeb28c41774efc6696516d43a2cabcb84b245072b3c8b2eee4dfaf3d5b

This is the strongest proof point. It verifies the end-to-end path:

Insta360 X4 -> Mac4 tmux recorder -> local session -> K11 archive/checksum/manifest

The previous 30-second screen-capture dry run also completed and uploaded to K11. That session was:

[home]/dev/lume-commerce-live/viz/lume-pcloud/docs/dance-sessions/20260527-184728-tier0-production-dry-run

It produced:

status: complete
labels: 3
video_bytes: 19005350
duration: 29.900001
codec: h264
frame: 1920x1080
pushed_to_k11: true
sha256: 26891a9874270db7f6cfb8f2c03ef22cf6c78d1c1eccc5a1191e7f7114de5069

This proved the recorder logic, label logic, inspection logic, optional MotionMix failure safety, and K11 transport on a longer duration before the physical-camera tmux fix was discovered.

An interrupted recording was tested and worked correctly. It preserved a partial MP4, preserved labels, and finalized with:

status: incomplete
incomplete: true
final_reason: interrupted by signal

This proves the crash-safety behavior is not merely theoretical.

Camera Failure Diagnosis

The main camera problem was subtle. AVFoundation camera listing succeeded. System Profiler saw the attached UVC devices:

Insta360 X4
Arducam B0478 (USB3 48MP)
Orbbec Femto Mega RGB Camera

macOS Camera privacy showed access enabled for:

Google Chrome
Insta360 Link Controller
Terminal
tmux
Unity

Yet direct recorder launches from Codex could see the devices and still get no frames. Chrome also repeatedly spawned `VideoCaptureService`, and old camera grid/browser preview work could wedge or claim the camera stack. Quitting Chrome and killing `cameracaptured` helped remove one source of contention, but direct Codex execution still produced zero frames. The decisive test was launching the same ffmpeg capture inside `tmux`, where it worked. Therefore the stable rule is:

For physical cameras on Mac4, use tmux-launched recorder commands.

If physical camera capture fails again, do not rewrite the recorder first. Run the recovery sequence:

pkill -f 'lume_rehearsal_camera_grid.py|run_lume_rehearsal_camera_grid.sh' 2>/dev/null || true
osascript -e 'tell application "Google Chrome" to quit' 2>/dev/null || true
killall cameracaptured 2>/dev/null || true
killall VDCAssistant 2>/dev/null || true
killall AppleCameraAssistant 2>/dev/null || true

Then re-run through `run_dance_recorder_tmux.sh` or `run_k11_weighted_take_tmux.sh`.

Do not use the browser grid as part of Tier 0 capture. The grid can be useful for alignment, but it can also keep camera services alive and confuse the capture path. The rehearsal recorder should own the camera when recording.

The Real Take Command

The current recommended command for the real weighted freestyle take is:

cd [home]/dev/lume-commerce-live/viz/lume-pcloud
tools/lume-dance-capture/run_k11_weighted_take_tmux.sh \
  --notes "weighted freestyle"

This expands to a tmux-launched recorder using:

camera: Insta360 X4
video size: 1920x1080
session name: full-weighted-take
expected duration: 3600 seconds
push K11: enabled

If the session should be named more specifically:

LUME_SESSION_NAME=weighted-freestyle-session-01 \
tools/lume-dance-capture/run_k11_weighted_take_tmux.sh \
  --notes "weighted freestyle, mocopi charging or unavailable, focus on wave/hold/torso labels"

If using Arducam instead of Insta360:

LUME_CAMERA_NAME="Arducam B0478 (USB3 48MP)" \
LUME_SESSION_NAME=weighted-freestyle-arducam-01 \
tools/lume-dance-capture/run_k11_weighted_take_tmux.sh \
  --notes "weighted freestyle with Arducam main review camera"

The recorder prints a session folder path. During the take, the operator should press:

1 for keep/signature moments
2 for wave/color moments
3 for high-energy burst moments
4 for torso lean or weight shift
5 for slow weighted hold
0 for still/recovery baseline
9 for bad/false-reactive moments
q to stop safely

If the shell is attached to the tmux recorder session, these keys go to the recorder. If the recorder is started detached and no one attaches, synthetic markers only happen when provided by CLI, so the real session should be attached or controlled through an operator UI that sends stdin.

Attach to the real recording session with:

/opt/homebrew/bin/tmux attach -t lume-dance-recorder

Why Not Direct Live Write To K11

It is tempting to record straight onto K11 because K11 has more storage. That should not be the first production route. A live network-only file path makes the only MP4 depend on an SSH/SCP pipe, Windows file behavior, Tailscale, Wi-Fi, sleep, and transient stalls. If anything in that chain breaks, the rehearsal could lose the only recording. The current route records locally first and pushes after finalization. That gives us an atomic local truth and a verified remote copy.

A future Tier 0.5 route could add redundant simultaneous recording:

ffmpeg local MP4
  plus
ffmpeg network mirror or post-segment upload

But that should be additive. The local MP4 should remain the authoritative recording until the network path has proven itself for long takes.

Disk Space Situation

Mac4 has been around 30 GB free during testing. The recorder warns below 50 GB and refuses below 20 GB unless `--force`. For a long weighted session, 30 GB is usable only if the take is modest and the bitrate remains controlled. It is not comfortable. A serious full session should ideally start with 80 to 100 GB free.

At the current default bitrate:

8 Mbps video
rough estimate: about 3.9 GB per hour before practical overhead variance

That sounds small, but the system has other live components, logs, caches, Unity artifacts, and repeated tests. Disk pressure also makes the computer feel slower. Before the real take, clean old Unity caches, old dry-run sessions, browser caches if needed, and stale camera-grid outputs. Do not delete successful K11-uploaded sessions blindly until their remote archive and manifest have been indexed or intentionally retained.

Optional MotionMix Telemetry

MotionMix capture lives at:

tools/lume-dance-capture/motionmix-capture/capture_motionmix.py

It is optional. It should be started beside a session only when it is useful. It writes:

raw/motionmix/motionmix_state.jsonl
raw/motionmix/devices.json
logs/motionmix-capture.log

It can target:

http://[ip]:9404
http://[ip]:9404

The important behavior is that MotionMix can be down without invalidating the take. A bad-hub test wrote an unavailable row and continued. That is exactly correct. MotionMix/iPhones are evidence, not the foundation of Tier 0.

The eventual high-value role for MotionMix is multi-angle evidence. The iPhones can provide side/rear/front pose or framing context. They should be synchronized by timestamps and markers, not forced into the live camera capture path until stable.

Unity, DYK, Femto/Mega, and Mocopi

Unity DYK should not be required for Tier 0 recording. The visual system can run independently or be off. The recorder is deliberately not a Unity capture engine. This separation protects the rehearsal data from Unity freezes, WebGL experiments, and visual performance spikes.

Femto/Mega remains the visible body aesthetic source for live DYK. It provides the real cyan performer, body mask, pCloud/point cloud, matte, depth texture, and body-shaped visual surface. The visual identity should come from the real body, not a fake mocopi skeleton or avatar. Mocopi is the motion backbone, not the visual performer.

Sony mocopi was noted as off/charging in the recent working context. That is fine for Tier 0. The recorder can collect useful weighted freestyle video and labels without mocopi. When mocopi returns, Unity state capture can add motion features and template evidence. The pipeline should not wait for mocopi to be charged before collecting video labels.

The Web Fluid Lab remains an experimental rendering branch. It should not own the rehearsal capture path. It is valuable for exploring body-mask-driven fluids, distortion, silhouette edge behavior, and mode grammars. But the recording of training evidence should be plain and reliable.

Computational Choreography Implication

The rehearsal capture stack is the first practical substrate for computational choreography. The body is not merely a tracking source. It is source material. The session video, labels, optional Unity state, optional mocopi, optional MotionMix/iPhone telemetry, and later SAM3D reconstructions are all different projections of the same embodied event.

This is where LIM-RPS enters concretely. LIM-RPS means Lipschitz-constrained implicit map for recursive polymodal synthesis. In practical LUME terms, the movement-to-visual map should be expressive but bounded. Small changes in body state should not cause unbounded visual explosions. Stillness should converge toward calm. A burst should have a controlled envelope and cooldown. A gesture should trigger a distinct event without flicker. Multiple modalities can recursively influence a visual field, but their influence must remain stable enough for a performer to learn and repeat.

Tier 0 labels are the first primitive form of choreography supervision. Pressing `2` during a wave/color moment does not train a neural network. It creates a time anchor. Later, the analyzer can examine the two seconds before and one second after that marker, compare it against still/recovery and bad/false-reactive labels, and derive thresholds or templates. This is more valuable than rushing to ML because it makes the vocabulary legible. We need to know what the movement signatures are before asking a model to classify them.

The first template targets remain:

wave_color
burst_high_energy
torso_lean_weight_shift
weighted_slow_power_hold
still_recovery_baseline
bad_false_reactive

The first live trigger should still be wave -> color phase shift. But the route to make it good is:

record session
label waves/stillness/false-reactives
analyze windows
derive template thresholds
feed templates into Unity/Web Lab
tune visual response under LIM-RPS constraints
repeat

Analysis Roadmap

After a real take lands on K11, the next practical layer is analysis, not more live preview. The analyzer should read:

session.json
raw/video/webcam_main.mp4
raw/labels/labels.jsonl
raw/unity/unity_state.jsonl                 optional
raw/motionmix/motionmix_state.jsonl         optional
derived/sam3d/sam3d_frames.jsonl           optional

It should output:

derived/templates/gesture_templates.json
derived/templates/gesture_windows.jsonl
derived/templates/session_summary.md
derived/templates/timeline.csv

If only video and labels exist, the report should still be useful. It can list marker density, label timing, available streams, missing streams, and candidate windows. If Unity state exists, it can extract mocopi/body motion features. If SAM3D exists, it can add reconstructed body features. If MotionMix exists, it can provide multi-angle auxiliary evidence.

The first features should be simple:

total_motion_energy
left_motion_energy
right_motion_energy
left_right_asymmetry
upper_body_motion
torso_lean
hand_spread
burst_score
stillness_score
motion_onset_speed
motion_decay_time
repeated_pattern_confidence
label_source
source_streams_available

No neural network should be trained until there are repeated labeled sessions with stable examples, multiple perspectives where possible, negative labels, and known failure cases. False-reactive labels are essential. They teach the system what not to trigger on.

K11 Storage Contract

K11 should store the incoming archive, checksum, and manifest. The storage root remains:

C:\lume\dance-sessions

Expected subfolders:

C:\lume\dance-sessions\_incoming
C:\lume\dance-sessions\_manifests
C:\lume\dance-sessions\_processed
C:\lume\dance-sessions\_exports
C:\lume\dance-sessions\_logs

The upload does not automatically extract. Extraction and indexing should be a separate K11 storage-management step, ideally idempotent and checksum-aware. Incoming archives should not be deleted automatically.

The K11 bridge and Rekordbox path remain out of scope:

Do not touch C:\temp\lume_rekordbox_bridge.py
Do not restart MotionMixLumeBridge
Do not send keyboard commands
Do not send MIDI commands
Do not make visual systems command Rekordbox

This separation is essential. K11 can store dance sessions while still being the final DJ safety gate for AirDeck/Rekordbox. Storage and DJ control are separate responsibilities.

Operator Runbook

Before the session:

1. Confirm disk has enough free space, ideally 80 to 100 GB.
2. Close camera preview/grid/browser camera tabs.
3. Confirm Insta360 X4 is connected.
4. Confirm K11 is reachable if upload is required.
5. Start the recorder through the K11 tmux launcher.
6. Attach to the tmux session if live keyboard labels will be pressed.

Start:

cd [home]/dev/lume-commerce-live/viz/lume-pcloud
tools/lume-dance-capture/run_k11_weighted_take_tmux.sh \
  --notes "weighted freestyle"

Attach:

/opt/homebrew/bin/tmux attach -t lume-dance-recorder

Stop:

Press q in the recorder terminal.

Inspect:

python3 tools/lume-dance-capture/lume_dance_recorder.py \
  --inspect-session docs/dance-sessions/YYYYMMDD-HHMMSS-session-name

Retry K11 upload:

tools/lume-data-transport/push-session-to-k11.sh \
  docs/dance-sessions/YYYYMMDD-HHMMSS-session-name

If the camera fails:

The session should finalize incomplete.
The labels and logs should remain.
Read logs/recorder.log and logs/ffmpeg.log.
Reset camera services.
Use the tmux launcher.
Do not switch back to browser camera preview during a take.

Current Risk Register

The largest immediate risk is disk space. Mac4 is too close to full for comfort. Even if the bitrate math says an hour is manageable, system performance and capture reliability degrade when the disk is under pressure. Free space should be improved before a serious full session.

The second risk is camera ownership. Chrome, camera grids, and preview tools can revive `VideoCaptureService` and interfere with capture. The real take should avoid browser camera pages. If alignment is needed, do it before the take, then quit the preview tools and start the recorder.

The third risk is operator labeling. If the tmux recorder is detached and no one attaches, live keyboard labels will not be pressed. The future operator monitor should control the tmux recorder or pipe marker keys into stdin. For now, attach to the tmux session for real labeling.

The fourth risk is assuming K11 upload means analysis is done. Upload only stores the session. It does not index, extract, analyze, reconstruct, or train. Those remain downstream phases.

The fifth risk is returning to overly complex live capture before Tier 0 produces a real session. The system should resist that. Get one full weighted take first.

Immediate Next Actions

The next best action is not another visual experiment. It is a full or medium-length weighted rehearsal capture using the proven K11 route. The purpose of that take is to collect real video and labels.

Minimum useful take:

10 to 15 minutes
Insta360 X4
labels for wave, hold, torso, burst, still, false-reactive
upload to K11
inspect complete status

Ideal first full take:

30 to 60 minutes
weighted freestyle
repeated wave/color attempts
slow weighted holds
torso leans and shifts
deliberate still baselines
intentional false-reactive labels
K11 upload verified

After that:

1. Run K11 indexer on incoming archives.
2. Copy or mount the session for Mac5 offline SAM3D sampling.
3. Run the template analyzer.
4. Generate gesture_templates.json.
5. Feed wave_color template back into Unity/Web Lab.
6. Tune wave -> color shift under LIM-RPS constraints.
7. Repeat with stronger labels.

Final Position

The system is now in a healthier tiered posture. Mac4 can capture the body with the Insta360 through tmux. The session can survive interruption. K11 can receive verified storage artifacts. MotionMix can be absent without breaking the take. Unity can be on or off without owning the recording. Mocopi can be charging without blocking data collection. Mac5 can remain offline analysis. Rekordbox stays untouched.

The major shift is conceptual as much as technical: LUME now has a capture spine. That spine is plain enough to trust. The next Duncan-level leap will not come from adding one more live effect before data exists. It will come from recording the body, labeling the body, mining the body, and then letting the visual system learn which movements deserve signature responses.