LUME K11/Mac4/Mac5 Pose Coach Capture Convergence Plan

LUME K11/Mac4/Mac5 Pose Coach Capture Convergence Plan

Date: 2026-05-28

Related source document:

Docs/LUME_REHEARSAL_CAPTURE_OVERWHELMING_UPDATE_2026-05-28.md

Decision

The current conversion should converge exactly three machines:

Mac4 -> long-take capture and live visuals
K11  -> durable storage, Pose Coach, AirDeck, Rekordbox safety
Mac5 -> offline reconstruction and heavy body analysis

Mac3 is not part of this live capture architecture. If Mac3 later produces
camera, sensor, or analysis data, it should enter as another source adapter into
the same rehearsal bundle contract. It should not create a fourth primary
architecture.

Core Conversion

The new architecture is not "Pose Coach versus Tier 0 recorder." It is:

one rehearsal bundle
  with multiple evidence sources
  stored durably on K11
  analyzed offline by Mac5
  and promoted back into Mac4 Unity / K11 AirDeck only after review

Mac4's tmux Tier 0 recorder is now the reliable long-take evidence spine. It
owns the continuous rehearsal video, the coarse marker labels, and the local
capture-first guarantee.

K11 Pose Coach is the guided short-clip and annotation layer. It owns the
operator-facing body framing view, AirDeck zone overlay, short gesture clips,
raw/overlay videos, keyframes, hand/body/cursor metadata, and promotion
candidate evidence.

Mac5 is the offline reconstruction worker. It should sample completed video
after the take, run SAM3DBody-cpp, and write derived 3D/body artifacts back into
the same session structure. It should not be in the live performance path.

K11 remains the only machine that may send Rekordbox commands. Mac4 Unity,
MotionMix, mocopi, Mac5 reconstruction, and raw camera streams must not directly
dispatch keys or MIDI.

Why This Matters

Right now there are several partially overlapping capture systems:

Mac4 Tier 0 recorder
  records long rehearsal video and coarse markers

K11 Pose Coach
  records short training clips, overlays, body boxes, hand cursors, and AirDeck labels

K11 AirDeck bridge
  turns promoted gestures into guarded Rekordbox commands

Mac5 SAM3D worker
  reconstructs body evidence from completed videos

MotionMix / mocopi / phones
  optional body-state evidence when available

They should not be merged by making one program own everything. That would make
the system brittle. The convergence point should be a shared manifest and folder
contract.

The correct unit is:

LUME rehearsal bundle

That bundle can contain one long Mac4 take, many K11 Pose Coach clips, optional
MotionMix/body-truth telemetry, optional mocopi, optional phone/watch streams,
and Mac5 derived reconstruction.

Machine Responsibilities

Mac4

Mac4 owns:

• Insta360/Arducam/Femto/Mega physical camera capture.
• tmux-launched Tier 0 long recording.
• `docs/dance-sessions/<session>/raw/video/webcam_main.mp4`.
• coarse live marker labels:
• `keep_signature`
• `wave_color`
• `burst_high_energy`
• `torso_lean_weight_shift`
• `weighted_slow_power_hold`
• `still_recovery_baseline`
• `bad_false_reactive`
• Unity DYK / pCloud visuals.
• local mocopi-to-Unity feed when sensors are charged.
• optional MotionMix capture beside a take.

Mac4 should not own:

• K11 Rekordbox key dispatch.
• final long-term storage.
• heavy offline reconstruction.
• the only copy of a session after upload.

Current verified Mac4 facts:

project root:
[home]/dev/lume-commerce-live/viz/lume-pcloud

long-take recorder:
tools/lume-dance-capture/lume_dance_recorder.py

tmux launcher:
tools/lume-dance-capture/run_dance_recorder_tmux.sh

K11 routed launcher:
tools/lume-dance-capture/run_k11_weighted_take_tmux.sh

operator monitor:
tools/lume-dance-capture/operator-monitor/lume_rehearsal_operator.py

Mac4's disk is currently tight. It is usable for short tests and modest takes,
but a serious rehearsal should start with more free space. Treat 50 GB as a
minimum comfort line and 80-100 GB as the target.

K11

K11 owns:

• `C:\lume\dance-sessions` as the durable dance-session storage root.
• incoming archives/checksums/manifests:
• `_incoming`
• `_manifests`
• `_processed`
• `_exports`
• `_logs`
• Pose Coach recording under:
• `C:\lume\recordings\pose-coach`
• AirDeck live camera baseline and promoted gesture runtime.
• Rekordbox, loopMIDI, keyboard mapping, and command safety.

K11 should be treated as two separate responsibilities:

K11 storage responsibility
  accepts and indexes completed rehearsal bundles

K11 DJ safety responsibility
  decides whether a promoted gesture may command Rekordbox

Those responsibilities live on the same machine, but they must not collapse into
each other. Uploading a dance session should not restart the Rekordbox bridge.
Pose Coach recording should not bypass AirDeck promotion gates. Mac4 should not
use K11 storage access as permission to send DJ commands.

Current verified K11 facts:

C:\lume\dance-sessions exists
C:\lume\recordings\pose-coach exists
C:\lume\recordings\pose-coach\pose-coach-latest-session.json exists

The latest Pose Coach manifest uses:

schema: lume.pose_coach.training_session.v1
storage_model: per_session_folder_with_video_frames_metadata
metadata/training.jsonl
video/display-overlay.avi
video/camera-overlay.avi
video/raw-camera.avi
frames/display/*.png
frames/camera/*.png
frames/raw/*.png

That is already close to the structure needed for training and review. It should
be adapted into the rehearsal bundle instead of replaced.

Mac5

Mac5 owns:

• offline SAM3DBody-cpp reconstruction;
• frame sampling from completed rehearsal videos;
• derived body reconstruction artifacts;
• overnight/background analysis;
• conversion into template features that Mac4 Unity and K11 AirDeck can use.

Current verified Mac5 facts:

SAM3DBody-cpp binary:
[home]/sandbox/lume-body-reconstruction/SAM3DBody-cpp/build-mac-ort120/fast_sam_3dbody_run

worker:
[home]/sandbox/lume-body-reconstruction/lume-sam3d-worker/process_session.py

The intended Mac5 command remains:

python3 [home]/sandbox/lume-body-reconstruction/lume-sam3d-worker/process_session.py \
  --session /path/to/session \
  --fps 1

For label-first analysis:

python3 [home]/sandbox/lume-body-reconstruction/lume-sam3d-worker/process_session.py \
  --session /path/to/session \
  --label-windows-only \
  --fps 1

Mac5 should write derived artifacts back into:

derived/sam3d/

It should never modify the source MP4 or labels.

Unified Rehearsal Bundle Contract

The next architecture layer should introduce a single bundle format:

schema: lume.rehearsal_bundle.v1

Recommended folder layout:

rehearsal-bundles/<bundle_id>/
  bundle.json
  README.md

  sources/
    mac4_tier0/
      session.json
      raw/video/webcam_main.mp4
      raw/labels/labels.jsonl
      logs/

    k11_pose_coach/
      manifest.json
      metadata/training.jsonl
      video/display-overlay.avi
      video/camera-overlay.avi
      video/raw-camera.avi
      frames/

    motionmix/
      motionmix_state.jsonl
      devices.json

    mocopi/
      skeleton.jsonl

    phones/
      iphone_left/
      iphone_right/

    watches/
      watch_left/
      watch_right/

  derived/
    sam3d/
      sam3d_frames.jsonl
      summary.json
      worker.log

    templates/
      gesture_windows.jsonl
      gesture_templates.json
      timeline.csv
      session_summary.md

    airdeck/
      promotion_candidates.json
      physical_proof_matrix.json
      promoted_gestures.json

  reports/
    bundle_summary.md
    capture_quality.md
    storage_integrity.md

This structure lets the system save space because not every source needs a full
copy of every artifact. `bundle.json` should support references to existing
archive paths on K11 and local paths on Mac4/Mac5. Heavy data can stay where it
belongs while the manifest records checksums, byte sizes, timestamps, and
ownership.

Bundle Manifest Fields

Minimum `bundle.json`:

{
  "schema": "lume.rehearsal_bundle.v1",
  "bundle_id": "20260528-weighted-freestyle-01",
  "created_at_unix": 0,
  "primary_storage": {
    "machine": "K11",
    "root": "C:\\lume\\dance-sessions"
  },
  "sources": [],
  "derived": [],
  "labels": [],
  "machine_roles": {
    "mac4": "capture_visuals",
    "k11": "storage_posecoach_airdeck_rekordbox_gate",
    "mac5": "offline_reconstruction"
  },
  "safety": {
    "rekordbox_command_authority": "K11 only",
    "mac4_can_command_rekordbox": false,
    "mac5_can_command_rekordbox": false
  }
}

Each source entry should include:

{
  "source_id": "mac4_tier0",
  "source_type": "long_take_video",
  "machine": "Mac4",
  "path": "...",
  "archive_path": "...",
  "sha256": "...",
  "status": "complete",
  "timebase": "unix_seconds",
  "start_unix": 0,
  "duration_s": 0,
  "labels_path": "...",
  "notes": ""
}

Pose Coach source entries should include:

{
  "source_id": "k11_pose_coach_001",
  "source_type": "guided_pose_clip",
  "machine": "K11",
  "schema": "lume.pose_coach.training_session.v1",
  "manifest_path": "C:\\lume\\recordings\\pose-coach\\sessions\\...\\manifest.json",
  "training_jsonl_path": "metadata/training.jsonl",
  "review_video": "video/display-overlay.avi",
  "raw_camera_video": "video/raw-camera.avi",
  "coordinate_space": "normalized_camera_frame",
  "intended_use": [
    "airdeck_training",
    "pose_quality_review",
    "virtual_deck_follow_targets"
  ]
}

Mac5 derived entries should include:

{
  "derived_id": "sam3d_001",
  "source_id": "mac4_tier0",
  "machine": "Mac5",
  "tool": "SAM3DBody-cpp",
  "path": "derived/sam3d/summary.json",
  "sampling_policy": "label_windows_only_or_1fps",
  "status": "complete"
}

Marker and Label Convergence

Mac4 long-take labels and K11 Pose Coach/AirDeck labels should be mapped into a
single vocabulary.

Mac4 Tier 0 coarse labels:

keep_signature
wave_color
burst_high_energy
torso_lean_weight_shift
weighted_slow_power_hold
still_recovery_baseline
bad_false_reactive

K11 Pose Coach / AirDeck labels:

present_still
d1_play_pause
d2_play_pause
loop_toggle
scratch
sync
next_track
cue
safe_stop_rehearsal

Unified top-level categories:

presence
stillness
expressive_motion
weighted_motion
deck_transport
deck_loop
deck_sync
deck_scratch
negative_false_reactive
promotion_candidate
promotion_verified

The system should not force all labels into DJ commands. A `wave_color` marker
is for visual grammar first. A `d1_play_pause` label is for AirDeck command
promotion. Some body events can feed both later, but only after review.

Pose Coach Conversion

Pose Coach should not become the main long-take recorder. Its highest-value role
is:

coach + short-clip recorder + annotation surface + AirDeck geometry viewer

It should show:

- camera feed without stretching;
- body box and segmentation quality;
- hand cursors;
- AirDeck left/right deck zones;
- current label;
- recording state;
- whether the clip is evidence, calibration, negative, promotion candidate, or
promotion verified.

It should record:

• raw camera clip;
• overlay clip;
• display/dashboard clip;
• training JSONL;
• selected PNG keyframes;
• annotation metadata;
• follow targets:
• hand cursors;
• body anchor;
• body boxes;
• deck zone membership.

It should export a bundle source adapter:

Pose Coach session -> k11_pose_coach source entry

That adapter can be a small script first. It should read:

C:\lume\recordings\pose-coach\pose-coach-latest-session.json
C:\lume\recordings\pose-coach\sessions\<id>\manifest.json

and write:

C:\lume\dance-sessions\_processed\<bundle_id>\sources\k11_pose_coach\<id>/

or, for space savings, write only a manifest reference with checksum metadata.

Storage and Space Policy

The space-saving rule is:

copy manifests freely
copy heavy videos only when the target machine actually needs them
use checksums and references for everything else

Recommended ownership:

Mac4 local:
  active capture only, recent local sessions, camera logs

K11:
  canonical archives, checksums, manifests, processed bundle indexes

Mac5:
  temporary working copy for reconstruction, derived outputs, summaries

Do not keep redundant full MP4 copies on every machine forever. K11 should keep
the canonical archive. Mac5 should pull only the session being processed. Mac4
can clear old local sessions after K11 archive and manifest integrity are
verified and the session is no longer needed for immediate capture debugging.

Live Versus Offline Boundary

Live path:

Mac4 camera / Unity / optional mocopi
K11 Pose Coach / AirDeck camera baseline
K11 Rekordbox command gate

Offline path:

K11 stored bundle
Mac5 SAM3D reconstruction
template extraction
gesture review
promotion candidates

Promotion path:

labeled clips + proof matrix + dry-run/self-play + physical verification
  -> promoted AirDeck manifest
  -> K11 bridge may command Rekordbox

Mac4 Unity visual templates can move faster than Rekordbox command templates.
Visual response is lower risk. Rekordbox command promotion must remain stricter.

Implementation Waves

Wave 1 - Bundle Indexer on K11

Build an idempotent K11 script that:

• scans `C:\lume\dance-sessions\_incoming`;
• verifies `.sha256`;
• reads transport manifests;
• extracts archives to `_processed` only if not already extracted;
• writes or updates `index.json` and `index.jsonl`;
• never deletes incoming archives automatically.

Output:

C:\lume\dance-sessions\index.json
C:\lume\dance-sessions\index.jsonl
C:\lume\dance-sessions\_processed\<session>/

Wave 2 - Bundle Manifest Generator

Build a manifest generator that can create:

bundle.json

from a processed Mac4 Tier 0 session. It should be safe to rerun.

Wave 3 - Pose Coach Source Adapter

Build a K11 adapter that adds the latest or selected Pose Coach session into a
bundle as a `k11_pose_coach` source. It should not duplicate big video files
unless asked. Default should be reference + checksum.

Wave 4 - Label Alignment

Create the canonical label table mapping:

Mac4 marker labels
K11 Pose Coach labels
AirDeck gesture labels
negative/failure labels
visual-only labels
command-promotion labels

This table should become the source of truth for both Pose Coach UI and the
template analyzer.

Wave 5 - Mac5 Pull/Process/Return

Build or formalize the Mac5 handoff:

K11 _processed session -> Mac5 local working copy -> process_session.py -> derived/sam3d -> K11 _processed bundle

Start with label-window-only processing to control runtime and disk use.

Wave 6 - Template Analyzer

Build the first analyzer over:

• Mac4 labels;
• Pose Coach training JSONL;
• optional MotionMix/body-truth;
• optional Mac5 SAM3D.

First output:

derived/templates/gesture_windows.jsonl
derived/templates/gesture_templates.json
derived/templates/session_summary.md

First visual target:

wave_color -> Unity/Web Lab color phase shift

First AirDeck targets:

left_hand_raise -> d1_play_pause
right_hand_raise -> d2_play_pause
safe_stop_rehearsal

Scratch, loop, sync, and next-track should remain candidates until the proof
matrix has enough physical examples.

Wave 7 - Promote Back Into Runtime

Promote results separately:

visual templates -> Mac4 Unity / Web Lab
DJ command templates -> K11 AirDeck promotion manifest

This keeps expressive visuals fast and command control safe.

Immediate Next Build Tasks

1. Add the K11 bundle indexer.
2. Add `lume.rehearsal_bundle.v1` manifest generation for Mac4 Tier 0 sessions.
3. Add Pose Coach export adapter into the bundle.
4. Add label convergence table.
5. Add Mac5 pull/process/return script for one processed K11 session.
6. Run a smoke bundle using existing route-smoke sessions before recording a new
long take.

Current Readiness

Ready:

• Mac4 Tier 0 physical-camera capture through tmux.
• Mac4 to K11 archive/checksum/manifest upload.
• K11 dance-session storage root.
• K11 Pose Coach per-session recording contract.
• Mac5 SAM3DBody-cpp worker and binary.

Not ready:

• unified `lume.rehearsal_bundle.v1` manifest.
• automatic K11 archive extraction/indexing for new route-smoke sessions.
• Pose Coach session adapter into dance-session bundles.
• Mac5 automated return of derived outputs to K11.
• shared label vocabulary enforced across Mac4/Pose Coach/AirDeck.
• full AirDeck command promotion beyond baseline proven gestures.

Practical Operating Rule

For now:

Mac4 records the rehearsal.
K11 stores it and records guided Pose Coach clips.
Mac5 reconstructs it later.
K11 alone promotes gestures to Rekordbox.

That is the convergence. Everything else is a source, derived artifact, or
consumer of the same bundle.