Insta360 + K11/LUME Custom Camera Runbook

Insta360 + K11/LUME Custom Camera Runbook

Status: first integrated software path, 2026-05-23.

This runbook turns the Insta360 from a passive stream into a programmable
MotionMix/LUME source. The current production path uses MotionMixApp on iOS to
connect to the Insta360 SDK, extract virtual camera crops from the 360 preview,
run Vision pose detection, and publish LUME-compatible UDP JSON to K11 on
`[ip]:9705`.

Capability Audit

Official Insta360 developer docs say the SDK supports secondary development for
the X5, X4 Air, X4, X3, ONE RS 1-Inch, ONE RS, ONE X2, ONE R, and ONE X product
lines. Supported platforms are Windows, Linux, iOS, and Android. Each platform
SDK is split into Camera SDK and Media SDK.

Sources:

• SDK guide: `https://onlinemanual.insta360.com/developer/en-us/resource/sdk`
• iOS SDK: `https://github.com/Insta360Develop/iOS-SDK`
• Desktop CameraSDK C++: `https://github.com/Insta360Develop/Desktop-CameraSDK-Cpp`
• Desktop MediaSDK C++: `https://github.com/Insta360Develop/Desktop-MediaSDK-Cpp`
• OSC control surface: `https://github.com/Insta360Develop/Insta360_OSC`

Practical read for LUME:

- iOS SDK path: best current path because MotionMixApp already has the
SDK frameworks and the LUME UDP publisher.
- Desktop CameraSDK path: strong K11-native future path because the C++ SDK can
receive preview video/audio/gyro/exposure data over USB, but it requires a
Windows/Linux decoding and stitching service.
- MediaSDK path: useful for recorded `.insv` / `.insp` post-processing and
preview stitching, not required for today's live pose bridge.
- OSC path: useful for lightweight camera status/control experiments, but not
the primary live image path.

Current Local Assets

MotionMixApp already contains the required Insta360 iOS framework bundle:

• `Frameworks/INSCameraSDK.xcframework`
• `Frameworks/INSCameraServiceSDK.xcframework`
• `Frameworks/INSCoreMedia.xcframework`
• `Frameworks/SSZipArchive.xcframework`

The app imports the SDK conditionally in:

- `MotionMixApp/MotionMixApp-Bridging-Header.h`

The app integration lives in:

• `MotionMixApp/Services/Insta360Service.swift`
• `MotionMixApp/Services/PoseService.swift`
• `MotionMixApp/Services/JointDataUploader.swift`
• `MotionMixApp/MotionMixApp.swift`
• `MotionMixApp/Views/SettingsView.swift`

The K11/LUME UDP contract lives in:

- `Comp-Core/core/audio-media/cc-echelon/docs/LUME_AGENT_CONTRACTS.md`

Architecture

Insta360 camera
  -> iPhone Wi-Fi connection through Insta360 SDK
  -> MotionMixApp `Insta360Service`
  -> equirectangular preview frame
  -> virtual 640x480 crop, for example `insta360-front-wide`
  -> `PoseService` Vision body pose
  -> `LumeSensorPublisher`
  -> UDP JSON `source=insta360_virtual` to K11 `[ip]:9705`
  -> K11 LUME bridge
  -> Rekordbox / loopMIDI / Unity / stage logic

Source tagging is now explicit:

- built-in phone camera frames publish as `source=iphone_motionmix`
- Insta360 virtual crops publish as `source=insta360_virtual`
- the `role` field identifies the phone placement or crop, such as
`phone_front`, `phone_side_left`, `phone_side_right`, `phone_torso`, or
`insta360-front-wide`
- the optional `camera` object carries virtual camera metadata: width, height,
yaw, pitch, FOV, display name, and virtual camera id

K11 can therefore react differently to a fixed phone camera, a 360 master-wide,
or a future fused source.

For the current live DJ rig, do not let multiple raw cameras drive Rekordbox
controls at the same time. Keep Bolt/webcam as the accepted live source and let
MotionMix/iPhone be a recorded assist source until a fuser emits a single
`source=fused` stream.

Mac2 USB/UVC Reconstruction Angle Lane

When the Insta360 is physically connected to Mac2, treat Mac2 as a camera host
for reconstruction evidence, not as a K11 replacement. The lane is:

Insta360 on Mac2
  -> Mac2 local angle recorder
  -> Desktop/MotionMix/mac2-angle-sessions
  -> optional K11 storage handoff
  -> Mac5 reconstruction after the Mac4-first gate allows it

The source identity is:

source_id: mac2_insta360_room_wide
host: mac2
role: reconstruction_room_wide
authority: evidence_only
command_authority_changed: false

Use the Mac2 probe before capture:

Desktop/MotionMix/mac2-insta360/probe_mac2_insta360.sh

The probe checks macOS camera devices, USB hints, ffmpeg AVFoundation devices,
and OpenCV indices. If only `FaceTime HD Camera` appears, the Insta360 is
attached but not exposing a capture feed to macOS yet. Switch the Insta360 into
webcam/UVC mode or start the Insta360 desktop bridge, then run the probe again.

Once the probe prints the real AVFoundation camera name, run:

LUME_CAMERA_NAME="Insta360 X4" \
LUME_SECONDS=8 \
Desktop/MotionMix/mac2-insta360/run_mac2_insta360_angle_tmux.sh

If the device appears under another name, use the exact printed name. The
recorder intentionally avoids silent OpenCV index fallback unless
`LUME_ALLOW_OPENCV_FALLBACK=1` is set, so it does not label the built-in Mac2
FaceTime camera as the Insta360 angle by mistake.

Each Mac2 session writes `derived/reconstruction/device_topology.json` with
`k11_only_command_gate=true`. K11 can store or compare this evidence later, but
K11 must not accept `mac2_insta360_room_wide` as live command evidence.

iPhone Setup

1. Connect the iPhone to the Insta360 camera Wi-Fi.
2. Open MotionMixApp.
3. Go to Settings.
4. In `360 Camera`, enter the Insta360 SDK app id and secret key.
5. Tap `Connect via WiFi`.
6. Tap `Start 360 Preview`.
7. Enable `Multi-Camera Mode`.
8. Leave `Front Wide` enabled for the first K11 test.
9. In `Apartment Stage`, enable `LUME Sensor Mode`.
10. Confirm:
- host: `[ip]`
- port: `9705`
- source: `iphone_motionmix` is okay for phone camera frames; Insta360
frames override this to `insta360_virtual`
- role: set the physical placement, for example `phone_side_left`,
`phone_side_right`, `phone_front`, or `phone_torso`
- stage: current room/stage id

No-Hardware Test Mode

MotionMixApp has a built-in synthetic 360 preview:

1. Open Settings.
2. Go to `360 Camera`.
3. Tap `Test Mode (No Hardware)`.
4. Enable `LUME Sensor Mode`.
5. Move in front of the iPhone camera if using the regular camera path, or use
the synthetic preview path to verify preview/reprojection UI and stream state.

Test mode proves app wiring, source tagging, and LUME packet construction. It
does not prove the live Insta360 Wi-Fi connection.

For a K11-side simulation without the iPhone, use the replay tool:

python [home]/Desktop/Comp-Core/core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/pose_replay.py \
  --target [ip]:9705 \
  --motion sweep \
  --rate 20 \
  --duration 30 \
  --camera-profile insta360-front-wide

To inspect the exact JSON before sending:

python [home]/Desktop/Comp-Core/core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/pose_replay.py \
  --camera-profile insta360-front-wide \
  --dry-run

K11 Verification

On K11, the bridge should already listen on UDP `[ip]:9705`.

Expected packet traits:

{
  "source": "insta360_virtual",
  "role": "insta360-front-wide",
  "w": 640,
  "h": 480,
  "landmarks": [ "... 33 BlazePose-style entries ..." ],
  "camera": {
    "virtual_camera_id": "insta360-front-wide",
    "display_name": "Front Wide",
    "yaw": 0,
    "pitch": 0,
    "fov": 110
  }
}

Quick checks:

Get-NetFirewallRule -DisplayName "LUME pose UDP 9705"
Get-NetUDPEndpoint -LocalPort 9705

If packet counts do not increase:

1. Confirm Tailscale is online on both devices.
2. Confirm K11 is still `[ip]`.
3. Confirm Windows Defender still allows UDP `9705` from `[ip]/10`.
4. Confirm MotionMixApp shows LUME packets sent increasing.

K11 Body Motion Database

The live K11 bridge now records the motion evidence stream locally:

• DB path: `C:\lume\data\body_motion.sqlite3`
• Session table: `motion_sessions`
• Frame table: `pose_frames`
• Event table: `pose_events`
• SAN training sidecar DB: `C:\lume\data\body_motion_san.sqlite3`
• SAN receiver endpoint: `POST http://[ip]:9471/san-frame`
• SAN receiver task: `LumeSanReceiver`

The recorder captures every valid pose packet that reaches the bridge, including
filtered sources. Accepted Bolt frames store raw JSON plus derived bridge
features: energy, arms height, hand spread, hip sway, body height, hand-raise
score, and head-nod value. Filtered packets are kept as raw evidence so later
fusion work can compare sources instead of throwing them away.

This means the current system is no longer just a live controller. It is also a
body-motion ledger that can be used to learn Mohamed's seated, standing, torso,
hand, head, and room-position patterns over time.

MotionMixApp's SAN trajectory logger now streams training frames directly to
K11 over Tailscale while keeping the phone-local JSONL backup in
`Documents/san-training`. The K11 SAN receiver stores those frames in the
sidecar SQLite database instead of the live pose database so training capture
cannot block Rekordbox gesture control. Verify it with:

curl http://[ip]:9471/health
python C:\temp\lume_multisource_report.py --window-seconds 120

In the report, `SAN Training` rows from `body_motion_san.sqlite3` confirm the
iPhone is sending SAN frames to K11.

Labeling Motion Examples

The bridge also opens a local K11 UDP label-control port:

• Host: `[ip]`
• Port: `9710`
• Utility: `C:\temp\lume_motion_label.py`

Use labels to mark intentional examples while the pose stream records. The label
rows can later be joined against `pose_frames` by timestamp.

python C:\temp\lume_motion_label.py segment hand_raise --seconds 5
python C:\temp\lume_motion_label.py segment head_nod --seconds 4
python C:\temp\lume_motion_label.py segment seated_idle --seconds 10
python C:\temp\lume_motion_label.py segment torso_lean_left --seconds 5
python C:\temp\lume_motion_label.py mark bad_tracking

For a real hand-raise capture, stand or sit in the pose viewer until the coach
says it can see your face/shoulders/hand, run the `segment hand_raise` command,
raise one hand clearly for the segment duration, then lower it.

Guided Capture And Calibration

For repeatable captures, use the guided capture script. It speaks instructions
on K11 unless `--no-speak` is passed.

python C:\temp\lume_guided_capture.py --profile quick
python C:\temp\lume_guided_capture.py --profile full
python C:\temp\lume_guided_capture.py --sequence seated_idle:10,hand_raise:5,head_nod:4

Profiles:

• `smoke` - one short technical test label.
• `quick` - seated idle, hand raise, head nod, torso left/right, hands-visible idle.
• `full` - left/right hand raise, seated/standing idle, head nod, torso leans.

After capture, run calibration:

python C:\temp\lume_motion_calibrate.py

It reads `motion_labels` and `pose_frames`, prints per-label feature statistics,
and writes:

C:\lume\data\gesture_calibration.json

The calibrator should not be treated as a trained model yet. It proposes
thresholds only when there are enough baseline and positive examples. If it says
`insufficient_data`, collect more labeled examples instead of hand-tuning blindly.

Quick K11 check:

python -c "import sqlite3; con=sqlite3.connect(r'C:\lume\data\body_motion.sqlite3'); print(con.execute('select count(*) from pose_frames').fetchone())"

Multi-source check:

python C:\temp\lume_multisource_report.py --window-seconds 300

The expected first topology is:

- `orbbec_bolt:bolt-rgb` or the current K11 webcam/Bolt viewer: accepted for
live gestures and Rekordbox controls.
- `iphone_motionmix:phone_side_left` / `phone_side_right` / `phone_front`:
recorded as a second angle from MotionMix on iPhone.
- `insta360_virtual:<crop>`: optional recorded or future fused source.
- `fused:*`: future canonical source after calibration/fusion, safe to accept
for live controls.

If the report shows more than one accepted raw source, fix the bridge
`--accept-source` flags before using live gestures.

Implementation Notes

The first integrated path is intentionally source-compatible with the existing
LUME schema. It does not require a new K11 bridge. Unknown fields are already
ignored by the bridge, so the added `camera` metadata is safe.

Files changed for this path:

• `MotionMixApp/Models/LatentState.swift`
• `MotionMixApp/Services/PoseService.swift`
• `MotionMixApp/Services/Insta360Service.swift`
• `MotionMixApp/Services/JointDataUploader.swift`
• `MotionMixApp/MotionMixApp.swift`
• `Comp-Core/core/audio-media/cc-echelon/docs/LUME_AGENT_CONTRACTS.md`
• `Comp-Core/core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/pose_replay.py`

Limitations

- The iOS SDK preview path depends on the phone staying connected to the
Insta360 Wi-Fi.
- iOS Wi-Fi connection to the camera may reduce normal internet routing; use
cellular/Tailscale behavior carefully during live operation.
- The current pose path publishes the first successfully extracted virtual crop,
not all enabled crops as independent K11 sources.
- The current implementation uses Apple's Vision pose detector, then maps the
available joints into the 33-entry LUME/BlazePose-shaped payload.
- Desktop USB preview on K11 is not implemented yet; it is the next robustness
upgrade if the iPhone Wi-Fi bridge is too fragile on-site.

Next Deployment Path

1. Build and install MotionMixApp on the iPhone.
2. Run `Test Mode (No Hardware)` with LUME Sensor Mode enabled and verify K11
packet receipt.
3. Connect the real Insta360 over Wi-Fi, start preview, and verify packets show
`source=insta360_virtual`.
4. Add a K11 bridge log line or dashboard filter that separates
`iphone_motionmix` from `insta360_virtual`.
5. If the live Wi-Fi path is unstable, build the K11-native desktop service:
Insta360 Desktop CameraSDK C++ over USB -> decode preview frames -> MediaPipe
pose -> same UDP schema.

Verification Completed

Completed on 2026-05-23:

python3 -m py_compile [home]/Desktop/Comp-Core/core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/pose_replay.py
git -C [home]/Desktop/MotionMixApp diff --check -- MotionMixApp/Models/LatentState.swift MotionMixApp/Services/PoseService.swift MotionMixApp/Services/Insta360Service.swift MotionMixApp/Services/JointDataUploader.swift MotionMixApp/MotionMixApp.swift
git -C [home]/Desktop/Comp-Core diff --check -- core/audio-media/cc-echelon/docs/LUME_AGENT_CONTRACTS.md core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/pose_replay.py core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/lume_rekordbox_bridge.py core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/run_lume_bridge_logged.cmd
xcodebuild -workspace [home]/Desktop/MotionMixApp/MotionMixApp.xcworkspace -scheme MotionMixApp -destination 'generic/platform=iOS' CODE_SIGNING_ALLOWED=NO build

Results:

- Python replay tool compile passed.
- Diff whitespace checks passed.
- MotionMixApp generic iOS build passed.
- Local UDP simulation received a valid packet with
`source=insta360_virtual`, `role=insta360-front-wide`, and 33 landmarks.

Deployment continuation on 2026-05-23:

- K11 `[ip]` was reachable over Tailscale.
- K11 had a UDP listener bound on `[ip]:9705` owned by
`pythonw.exe C:\temp\lume_rekordbox_bridge.py --midi-port LUME --rate-hz 30 --gesture-enable`.
- Signed MotionMixApp device build passed for physical iPhone with:
- bundle id `com.openclaw.MotionMixApp`
- version `1.0.0`
- build `2`
- signing identity `Apple Development: [email]`
- Install to iPhone 14 Pro Max `00008120-000128692244201E` was blocked by
insufficient device storage: about 94 MB required, about 36 MB available.
- Installed and launched the updated app on iPhone 16 Pro Max
`00008140-001818491A88801C`.
- Sent a 39-packet Insta360-profile replay burst to K11:

python3 [home]/Desktop/Comp-Core/core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/pose_replay.py \
  --target [ip]:9705 \
  --camera-profile insta360-front-wide \
  --motion sweep \
  --rate 20 \
  --duration 2

K11 source telemetry continuation on 2026-05-23:

- Patched K11 bridge source file:
`C:\temp\lume_rekordbox_bridge.py`
- Backup created:
`C:\temp\lume_rekordbox_bridge.py.bak-20260523-insta360-source`
- Local and K11 SHA-256 matched:
`18c383da5d1963b73310ff982ecaf525330ea03582f76c1bba3ead6f1d2a52d3`
- Added bridge heartbeat fields:
- `source=<latest-source[:role]>`
- `sources=<source[:role]=count,...>`
- Foreground K11 verification received replay packets and printed:
`source=mega sources=mega=50,insta360_virtual:insta360-front-wide=17`
followed by later counts including
`insta360_virtual:insta360-front-wide=136`.
- K11 bridge was relaunched through Task Scheduler as `MotionMixLumeBridge`.
- Persistent process after SSH disconnect:
`pythonw.exe C:\temp\lume_rekordbox_bridge.py --midi-port LUME --rate-hz 30 --gesture-enable`
- UDP listener remained bound on `[ip]:9705`.

Logged K11 field-test continuation on 2026-05-23:

- Added launcher:
`C:\temp\run_lume_bridge_logged.cmd`
- Launcher SHA-256:
`a0df32cb1137379335a483f0d737d5aada251d2e8a040a6eb29f9229a2134d3b`
- Repointed scheduled task `MotionMixLumeBridge` to:
`C:\Windows\System32\cmd.exe /d /c C:\temp\run_lume_bridge_logged.cmd`
- Hardened scheduled task:
- trigger: at logon for `NUCBOX_K11\Mohamed Diomande`
- restart count: `10`
- restart interval: `PT1M`
- The wrapper runs:
`python.exe C:\temp\lume_rekordbox_bridge.py --midi-port LUME --rate-hz 30 --gesture-enable`
- Logs now persist at:
- `C:\temp\lume_rekordbox_bridge.out.log`
- `C:\temp\lume_rekordbox_bridge.err.log`
- Persistent log verification:
- live traffic appeared as `source=mega`
- replay traffic appeared as `source=insta360_virtual:insta360-front-wide`
- source counts included `insta360_virtual:insta360-front-wide=135`
- Task stayed `Running` and UDP remained bound on `[ip]:9705`.
- After hardening, the restarted task wrote a fresh startup line and live
`source=mega` heartbeats to `lume_rekordbox_bridge.out.log`.

To watch the live field test from Mac:

ssh k11 'powershell -NoProfile -Command "Get-Content C:\temp\lume_rekordbox_bridge.out.log -Tail 80 -Wait"'

Bolt rollback and stable field state on 2026-05-23:

- Decision: use the Orbbec Femto Bolt as the active body-tracking source and
leave Insta360 X4 as webcam/OBS only for now.
- K11 Bolt devices are healthy:
- `Orbbec Femto Bolt RGB Camera`
- `Orbbec Femto Bolt Depth Camera`
- Restarted `LUME-BoltSkeleton`; it is `Running` and automatic.
- Active Bolt publisher:
`python.exe C:\lume\services\bolt-skeleton-pub\bolt_rgb_blazepose_pub_v2.py --host [ip] --port 9702 --fps 30 --record-ctrl-port 9788 --record-dir "C:\Users\Mohamed Diomande\Desktop"`
- Patched K11 Bolt publisher JSON to include:
- `source=orbbec_bolt`
- `role=bolt-rgb`
- K11 backup:
`C:\lume\services\bolt-skeleton-pub\bolt_rgb_blazepose_pub_v2.py.bak-20260523-orbbec-source`
- Deployed Bolt publisher SHA-256:
`28abc63172c4ab953b5493577a884cb3a2b0f48e5cd00d931f5fbc4fb8b54788`
- Patched the bridge to log sender peers and support source allow-listing.
- Repointed `C:\temp\run_lume_bridge_logged.cmd` to run:
`python.exe C:\temp\lume_rekordbox_bridge.py --midi-port LUME --rate-hz 30 --gesture-enable --accept-source orbbec_bolt:bolt-rgb`
- Bridge backups:
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-peer-telemetry`
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-source-filter`
- `C:\temp\run_lume_bridge_logged.cmd.bak-20260523-source-filter`
- Deployed bridge SHA-256:
`23510869ee18d5d54418836e452a52bd98bc0e8b5bab17d3db472fd9da0fbd13`
- Deployed launcher SHA-256:
`120550f204ef5fbacc904d1c4a6423c60f8c5bbee7d5a11835c1d73313e2ffc7`
- Stale Mac4 publisher still exists but is filtered:
`mac4_pose_udp_publisher.py --pose-host [ip] --pose-port 9705 --source mega`
- Source-filter verification:
- accepted: `orbbec_bolt:bolt-rgb@[ip]`
- filtered: `mega@[ip]:59989`
- bridge log showed `source filter = ['orbbec_bolt:bolt-rgb']`
- bridge frames continued from the Bolt source only.
- Rekordbox play gesture verification:
- raw interactive test focused Rekordbox and sent `Z`
- patched bridge play gesture from `SPACE` to `Z`
- deployed backup: `C:\temp\lume_rekordbox_bridge.py.bak-20260523-play-z`
- deployed backup after hot-cue split:
`C:\temp\lume_rekordbox_bridge.py.bak-20260523-z-no-hotcue`
- bridge log confirmed `GESTURE arms_up -> Z (play/pause)`
- added `--hotcue-enable` flag and left it disabled in the launcher, so
energy spikes no longer send `Q` during play/pause testing.

Hand play/pause gesture hardening on 2026-05-23:

- Root cause of the unreliable gesture: the bridge was treating Bolt/MediaPipe
world landmarks like normalized image coordinates, and it sent keyboard
shortcuts to whichever window had focus.
- Patched the bridge so the Rekordbox gesture:
- uses relative wrist-vs-shoulder geometry instead of absolute arm height
- accepts either hand as the play/pause gesture
- requires a lowered-hand hold before arming
- requires a raised-hand hold before firing
- focuses the Rekordbox window before sending `Z`
- Current gesture contract:
1. Lower either hand below its shoulder briefly.
2. Raise that hand clearly above its shoulder and hold for a beat.
3. Bridge focuses Rekordbox and sends `Z`, toggling play/pause.
- Current K11 bridge command remains:
`python.exe C:\temp\lume_rekordbox_bridge.py --midi-port LUME --rate-hz 30 --gesture-enable --accept-source orbbec_bolt:bolt-rgb`
- Current K11 bridge SHA-256:
`7b7f681b0613df67493065f175880558c9eb1606042601105dc73a374870f55a`
- Bridge backups from this tuning pass:
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-relative-arm-gesture`
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-rekordbox-focus`
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-arm-threshold-focus`
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-require-lowered-arm`
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-arm-hold-frames`
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-low-high-hold`
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-right-hand-only`
- `C:\temp\lume_rekordbox_bridge.py.bak-20260523-either-hand-raise`
- Final verification:
- `python3 -m py_compile` passed locally.
- K11 `python -m py_compile C:\temp\lume_rekordbox_bridge.py` passed.
- Local and K11 SHA-256 matched.
- Clean bridge restart stayed running on UDP `[ip]:9705`.
- Synthetic lowered-then-raised packet logged:
`GESTURE hand_raise -> Z (play/pause) ... focused=True foreground='rekordbox'`.
- Live retest showed the Bolt sees movement (`energy` up to `0.89`), but
the user hand raise did not cross the shoulder-relative trigger
(`armScore` stayed near `0.00`), so camera framing/hand height is the
remaining field issue.

Head-bow play/pause gesture on 2026-05-23:

- Added a nod/bow detector because the Bolt sees head/torso motion more
reliably than wrist-over-shoulder movement in the current room framing.
- Current default gesture behavior:
- `--gesture-enable` enables head bow play/pause.
- Hand raise is OFF unless the bridge is launched with
`--hand-raise-enable`.
- Hot cue `Q` remains OFF unless launched with `--hotcue-enable`.
- Current physical gesture:
1. Look forward for a beat.
2. Bow/nod head clearly downward once.
3. Bring head back up before trying again.
- The detector is intentionally strict:
- warmup frames prevent startup toggles;
- bow must cross `nod >= 0.20` for multiple frames;
- head must return below `nod <= 0.10` before re-arming.
- K11 bridge backup:
`C:\temp\lume_rekordbox_bridge.py.bak-20260523-head-bow-strict`
- Current K11 bridge SHA-256:
`4012b12e0bfd473a222656334c1fbfdda30e00e1579c480971a9fc91a2d5227c`
- Verification:
- local and K11 Python compile passed;
- local/K11 SHA-256 matched;
- bridge log confirms `hand raise keyboard = OFF` and `hot cue keyboard = OFF`;
- fast synthetic bow produced exactly:
`GESTURE head_nod -> Z (play/pause) nod=0.280 delta=0.080 focused=True foreground='rekordbox'`;
- subsequent live frames did not false-fire.

Voice keyboard shortcuts on 2026-05-23:

- Deployed voice shortcut dispatcher to K11:
`C:\temp\voice_dj.py`
- Scheduler task:
`LumeVoiceDJ`
- Runtime:
- mic: `Microphone (Realtek(R) Audio)`
- model: `gpt-realtime-2`
- log: `C:\lume\build\voice-dj.log`
- current verified pid after gesture-coach recovery: `33156`
- Current SHA-256:
`974b0fc0f96dc7c83a45d33558531360e50b3fa019f0f926523ed02cb77eda94`
- Supported voice shortcuts:
- `press Z` -> Deck 1 play/pause
- `press N` -> Deck 2 play/pause
- `press control F`
- `press shift A`
- `press alt A`
- `press hot cue one/two/three/four` -> keys `1/2/3/4`
- Semantic DJ commands still work:
- `play`, `pause`, `cue`, `next track`, `volume up`, `volume down`
- default deck is 1 unless the utterance names deck two/right.
- Safety hardening:
- every shortcut focuses Rekordbox before sending keys;
- tool calls now wait for the transcript before executing;
- transcript guard blocks near-homophones like `Peace.` from firing `pause`;
- bare one-letter audio like `Z` is ignored unless spoken as an explicit shortcut phrase such as `press Z`.
- Verification:
- local and K11 Python compile passed;
- K11 task restarted cleanly and stayed alive;
- guardrail check returned:
`peace_pause False`, `pause_pause True`, `press_z True`, `bare_z False`.

Gesture pose coaching on 2026-05-23:

- Added a spoken pose coach to the K11 bridge so the system can tell Mo where
to stand instead of relying on guesswork.
- Current bridge command:
`python.exe C:\temp\lume_rekordbox_bridge.py --midi-port LUME --rate-hz 30 --gesture-enable --coach-enable --coach-interval 6 --accept-source orbbec_bolt:bolt-rgb`
- Current bridge SHA-256:
`ef009860b7bc5fd5f03af2d4b2877d12cf6379a98e26297ea85e76d2cff638e9`
- Current launcher SHA-256:
`b04d88d76ab82a78c7d3bf3d833461460412d38a241c8106980276a2f6670908`
- Coach prompts are generated from Bolt pose landmarks:
- unclear body view -> `I need a clearer body view. Step into the camera frame.`
- too close/cropped -> `Move back a little. I need your head, hands, and hips in frame.`
- too far -> `Move closer. You are too far away for reliable gestures.`
- off-center -> `You are left/right of frame. Move toward the camera center.`
- hands hidden -> `Show both hands to the camera. I need your wrists for hand gestures.`
- good lock -> `I see you. Good position. Bow your head to play or pause.`
- Runtime state after recovery:
- `MotionMixLumeBridge` pid `42312`
- `LUME-BoltSkeleton` pid `32784`
- `LUME-Audio` pid `44324`
- `LumeVoiceDJ` pid `33156`
- Verification:
- local Python compile passed;
- K11 Python compile passed;
- bridge startup log shows `voice coach = ON interval=6.0s`;
- after closing Windows Camera and restarting `LUME-BoltSkeleton`, Orbbec color
stream recovered at about 30 fps;
- bridge received fresh accepted frames from
`orbbec_bolt:bolt-rgb@[ip]`;
- coach spoke:
`Move back a little. I need your head, hands, and hips in frame.`

Seated pose viewer on 2026-05-23:

- Windows Camera is not the preferred diagnostic surface for the Bolt. It often
errors because the Bolt color stream can only be owned cleanly by one process,
and LUME services/viewers need that stream.
- Added a dedicated K11 visual app:
- source: `core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/lume_pose_viewer.py`
- launcher: `core/audio-media/cc-echelon/tools/lume-rekordbox-bridge/run_lume_pose_viewer.cmd`
- deployed runtime: `C:\temp\lume_pose_viewer.py`
- deployed launcher: `C:\temp\run-lume-pose-viewer.cmd`
- The app opens a `LUME Pose Coach` window on K11 and shows:
- the real Orbbec/Bolt RGB frame;
- pose skeleton overlay;
- hand skeleton overlay;
- person segmentation overlay;
- bounding box/framing state;
- segmentation quality in the status panel;
- live coach instruction text;
- spoken coach prompts.
- It runs in seated/upper-body mode by default:
- required: face, shoulders, and at least one hand;
- not required: full legs or always-visible hips;
- full-body mode is still available with `--framing full`.
- While the viewer is open, it replaces `LUME-BoltSkeleton` as the pose source:
- `LUME-BoltSkeleton` should be stopped to avoid camera ownership conflict;
- viewer publishes compatible pose JSON to `[ip]:9705`;
- bridge continues receiving `source=orbbec_bolt`, `role=bolt-rgb`.
- Segmentation update on 2026-05-24:
- deployed model: `C:\lume\services\bolt-skeleton-pub\selfie_segmenter.tflite`;
- viewer runs MediaPipe `ImageSegmenter` every two frames by default;
- viewer emits compact `segmentation` JSON, not full masks;
- bridge records segmentation columns in `C:\lume\data\body_motion.sqlite3`;
- health check:
`python C:\temp\lume_segmentation_report.py --window-seconds 120 --limit 3`.
- Current running state:
- viewer pid `62316`;
- bridge pid `63460`;
- audio publisher pid `44324`;
- voice dispatcher pid `33156`;
- background `LUME-BoltSkeleton` stopped while viewer owns camera.
- Hashes:
- viewer `1ceec8d3807797a7805c5f8ae686563016c82a8a537831ba20b51cca8ec03ff1`
- viewer launcher `3fdd856e3099aceb13969d461a0c0e4ddbd9f4c11142ff3fa4902c32c8c9756f`
- bridge `1c042afe0fd5a159fcf6d6c4122e8e883a879a560d745ea48395615b3ffd7095`
- bridge launcher `0d4ec621da265747d4ab03affa63e2c491e22dd90efc7cdb781085efa01b4bc1`
- Verification:
- local Python compile passed for viewer and bridge;
- K11 viewer log shows `[pose-viewer] running; press Q in the viewer to quit`;
- bridge log shows `voice coach = ON interval=6.0s framing=upper`;
- bridge accepted fresh frames from `orbbec_bolt:bolt-rgb@[ip]:50277`;
- coach now asks for `face, shoulders, and at least one hand`, not full body.

Aspect-safe viewer update on 2026-05-23:

- Fixed the stretched/squished camera display.
- Root cause: OpenCV `WINDOW_NORMAL` was stretching the portrait Bolt image
into a landscape/fullscreen window.
- Viewer now renders a fixed 16:9 dashboard canvas:
- portrait camera feed is aspect-fitted, not stretched;
- unused space becomes neutral pillarbox/status area;
- right panel shows mode, instruction text, FPS, and gesture hints;
- window uses OpenCV `WINDOW_KEEPRATIO` when available.
- Current viewer SHA-256:
`37b3c5b202a1595503fb02be73fb5bb4af235fa0b18e8275789c6ea1bd9479c2`
- Deployed to K11 and restarted `LumePoseViewer`.
- Verification:
- local/K11 viewer hashes match;
- local Python compile passed;
- bridge receives fresh frames from the restarted viewer peer
`orbbec_bolt:bolt-rgb@[ip]:54616`;
- coach reports seated/upper-body tracking, including
`Good seated position` and `Move closer`.