Motion Mix Live Director Mac4 Camera 7/8/9 Handoff

Motion Mix Live Director Mac4 Camera 7/8/9 Handoff

Date: 2026-06-14

This handoff is for the agent integrating the three new Mac4 camera lanes into Motion Mix Live Director and the Motion Mix Wall. No Live Director application code was changed in this pass.

Current Outcome

Mac4 now exposes three name-selected AVFoundation camera lanes over HTTP:

Verified status URLs:

http://[ip]:8087/status?[sensitive field redacted]
http://[ip]:8088/status?[sensitive field redacted]
http://[ip]:8089/status?[sensitive field redacted]

Stream and frame URLs exposed by the lane services:

http://[ip]:8087/mjpeg?[sensitive field redacted]
http://[ip]:8087/snapshot?[sensitive field redacted]

http://[ip]:8088/mjpeg?[sensitive field redacted]
http://[ip]:8088/snapshot?[sensitive field redacted]

http://[ip]:8089/mjpeg?[sensitive field redacted]
http://[ip]:8089/snapshot?[sensitive field redacted]

Files Added In This Repo

Camera lane implementation:

services/avfoundation-camera-lane/avfoundation_camera_lane.py
services/avfoundation-camera-lane/README.md

LaunchAgent templates:

launchagents/com.lume.mac4-camera7-insta360.plist
launchagents/com.lume.mac4-camera8-femto-mega.plist
launchagents/com.lume.mac4-camera9-femto-bolt.plist

Operational runbook:

docs/runbooks/mac4-camera7-camera8-lanes.md

Mac3 exploratory work is documented separately:

docs/runbooks/mac3-insta360-camera7.md
services/mac3-insta360-camera7/
launchagents/com.lume.mac3-insta360-camera7.plist

What Was Installed On Mac4

The reusable lane script was copied to:

[home]/Applications/AvfoundationCameraLane/avfoundation_camera_lane.py

The LaunchAgents were installed to:

[home]/Library/LaunchAgents/com.lume.mac4-camera7-insta360.plist
[home]/Library/LaunchAgents/com.lume.mac4-camera8-femto-mega.plist
[home]/Library/LaunchAgents/com.lume.mac4-camera9-femto-bolt.plist

Each lane probes AVFoundation with ffmpeg, selects the camera by name/regex, and refuses unwanted devices through exclude patterns. This is important because AVFoundation indices changed while testing; the integration should not depend on index `0`, `1`, or `2`.

Current Mac4 Verification

Run these from Mac1:

curl -fsS 'http://[ip]:8087/status?[sensitive field redacted]
curl -fsS 'http://[ip]:8088/status?[sensitive field redacted]
curl -fsS 'http://[ip]:8089/status?[sensitive field redacted]

Expected selected devices:

8087 -> selected.name = Insta360 X4
8088 -> selected.name = Orbbec Femto Mega RGB Camera
8089 -> selected.name = Orbbec Femto Bolt RGB Camera

LaunchAgent checks:

ssh mac4 'launchctl print gui/$(id -u)/com.lume.mac4-camera7-insta360'
ssh mac4 'launchctl print gui/$(id -u)/com.lume.mac4-camera8-femto-mega'
ssh mac4 'launchctl print gui/$(id -u)/com.lume.mac4-camera9-femto-bolt'

Logs:

/tmp/mac4-camera7-insta360.log
/tmp/mac4-camera7-insta360.err
/tmp/mac4-camera8-femto-mega.log
/tmp/mac4-camera8-femto-mega.err
/tmp/mac4-camera9-femto-bolt.log
/tmp/mac4-camera9-femto-bolt.err

Important Caveat

The `/status` endpoints are confirmed good and currently identify all three physical cameras correctly.

Frame capture still needs one final operator-side fix before treating the feeds as video-ready in production. During this pass, ffmpeg could enumerate the cameras and report supported formats, but `/snapshot` and `/mjpeg` timed out while trying to pull frames. The likely causes are:

1. macOS Camera permission for the ffmpeg binary has not been approved in the GUI session on Mac4.
2. One or more Orbbec processes may already own the camera stack.

Do not stop existing Mac4 depth/body-truth publishers unless the operator explicitly allows it. One active process observed during testing was:

[home]/lume-femto/runtime-demo/pointcloud_pub.py ... --raw-depth ...

The integration agent should therefore treat the lanes as "device discovered and addressable" first, then verify frame delivery before making them active director sources.

Live Director / Wall Code Inspected

Swift app:

[home]/Desktop/MotionMixLiveDirector/MotionMixLiveDirector/Networking/ServerClient.swift
[home]/Desktop/MotionMixLiveDirector/MotionMixLiveDirector/Models/Device.swift
[home]/Desktop/MotionMixLiveDirector/MotionMixLiveDirector/Views/DeviceGrid.swift

Rust/HTML multicam server and Wall:

[home]/Desktop/MotionMix/multicam-server/src/main.rs
[home]/Desktop/MotionMix/multicam-server/src/wall.html

Relevant existing contract:

GET  /devices
POST /register
GET  /device/:id/stream
GET  /device/:id/frame
GET  /device/:id/snapshot
GET  /wall

The Wall currently renders from `/devices`, then proxies video through `/device/:id/stream?mode=program` and still frames through `/device/:id/frame?mode=program`.

Specific code landmarks from the current tree:

[home]/Desktop/MotionMix/multicam-server/src/wall.html:519
  Layout buttons currently stop at 6-Up.

[home]/Desktop/MotionMix/multicam-server/src/wall.html:631
  layoutSlotCount() returns a maximum of 6.

[home]/Desktop/MotionMix/multicam-server/src/wall.html:649
  pickerDeviceIds() caps the picker at baseVisibleDeviceIds().slice(0, 6).

[home]/Desktop/MotionMix/multicam-server/src/wall.html:665
  defaultVisibleDeviceIds() also starts from baseVisibleDeviceIds().slice(0, 6).

[home]/Desktop/MotionMix/multicam-server/src/wall.html:746
  streamURL() maps a registered device to /device/:id/stream?mode=program.

[home]/Desktop/MotionMix/multicam-server/src/wall.html:754
  snapshotURL() maps a registered device to /device/:id/frame?mode=program.

[home]/Desktop/MotionMix/multicam-server/src/main.rs:2535
  RegisterPayload defines the server registration fields.

[home]/Desktop/MotionMix/multicam-server/src/main.rs:2690
  list_devices() returns the /devices payload and prunes stale entries after 180 seconds.

[home]/Desktop/MotionMix/multicam-server/src/main.rs:7546
  device stream fan-out cache begins here.

[home]/Desktop/MotionMix/multicam-server/src/main.rs:8267
  device_frame() is the Wall's single-frame fallback path.

[home]/Desktop/MotionMixLiveDirector/MotionMixLiveDirector/Models/Device.swift:21
  MMDevice is the Swift app's decoded /devices model.

[home]/Desktop/MotionMixLiveDirector/MotionMixLiveDirector/Networking/ServerClient.swift:46
  fetchDevices() reads /devices from the MotionMix server.

Recommended Integration Path

Prefer registering the Mac4 lanes into the existing multicam server on Mac1 instead of making the Wall directly fetch Mac4 lane URLs.

The existing MotionMix server already has the needed shape:

1. `/register` accepts `id`, `name`, `ip`, `port`, `token`, `stream_ready`, `model`, placement fields, and `recording_capable`.
2. `/devices` returns the registered devices.
3. `/device/:id/stream` fans out a registered upstream MJPEG stream.
4. `/device/:id/frame` waits for a cached frame and returns a single JPEG.

Recommended registration payloads:

{
  "id": "mac4-camera7-insta360",
  "name": "Mac4 Camera 7 Insta360 X4",
  "ip": "[ip]",
  "tailscale_ip": "[ip]",
  "port": 8087,
  "token": "mac4camera7",
  "stream_ready": true,
  "model": "Insta360 X4",
  "angle_deg": 90,
  "height_level": 2,
  "distance_m": 2.5,
  "recording_capable": false
}

{
  "id": "mac4-camera8-femto-mega",
  "name": "Mac4 Camera 8 Femto Mega",
  "ip": "[ip]",
  "tailscale_ip": "[ip]",
  "port": 8088,
  "token": "mac4camera8",
  "stream_ready": true,
  "model": "Orbbec Femto Mega RGB Camera",
  "angle_deg": 105,
  "height_level": 2,
  "distance_m": 2.5,
  "recording_capable": false
}

{
  "id": "mac4-camera9-femto-bolt",
  "name": "Mac4 Camera 9 Femto Bolt",
  "ip": "[ip]",
  "tailscale_ip": "[ip]",
  "port": 8089,
  "token": "mac4camera9",
  "stream_ready": true,
  "model": "Orbbec Femto Bolt RGB Camera",
  "angle_deg": 75,
  "height_level": 2,
  "distance_m": 2.5,
  "recording_capable": false
}

Example manual registration against the current server:

curl -X POST 'http://[ip]:9404/register' \
  -H 'Content-Type: application/json' \
  -d '{"id":"mac4-camera7-insta360","name":"Mac4 Camera 7 Insta360 X4","ip":"[ip]","tailscale_ip":"[ip]","port":8087,"token":"mac4camera7","stream_ready":true,"model":"Insta360 X4","angle_deg":90,"height_level":2,"distance_m":2.5,"recording_capable":false}'

Repeat for cameras 8 and 9. The production version should be a small keepalive sidecar because `/devices` prunes entries older than 180 seconds.

Keepalive Sidecar Requirements

Create either a Mac4-side or Mac1-side sidecar that:

1. Polls each lane `/status?[sensitive field redacted]`.
2. Sets `stream_ready` to `true` only when `status.ok == true` and frame delivery has been verified.
3. Posts `/register` every 10 to 20 seconds to `http://[ip]:9404/register`.
4. Keeps `recording_capable: false` until the lane can produce recording-result callbacks compatible with the MotionMix gate.

For first integration, it is acceptable to register with `stream_ready: false` to prove Wall list ordering and labels, then switch to true after `/device/:id/frame` succeeds.

Wall Changes Needed

The Wall screenshot shows `6 cams`, `SAVE 6/6`, and a six-tile grid. The code confirms this is not just cosmetic:

[home]/Desktop/MotionMix/multicam-server/src/wall.html

Areas to update:

1. Add a `9-Up` layout button near the existing `6-Up` control.
2. Add a `nine` case to `layoutSlotCount`.
3. Replace `pickerDeviceIds() -> baseVisibleDeviceIds().slice(0, 6)` with a dynamic picker cap or `slice(0, 9)`.
4. Replace manual/hidden layout caps that currently slice to 6.
5. Replace `defaultVisibleDeviceIds()` base cap from `slice(0, 6)` to layout-aware capacity.
6. Expand keyboard shortcuts from `1` through `6` to `1` through `9`.
7. Update the Cuts panel title from static `1-6` to reflect the rendered range.
8. Update `#mini` CSS from `repeat(6, ...)` to a responsive 9-camera strip or two-row layout.
9. Decide whether 6-up remains the default with a picker, or whether adding Mac4 lanes should default to 9-up.

The current Wall already computes column count dynamically; counts over 4 become a 3-column grid, which works naturally for 9 cameras as 3x3.

Director Behavior To Preserve

Do not accidentally make these Mac4 lanes iOS-control targets.

The server currently considers director-eligible sources to be iPhone/iPad only. That means the Mac4 lanes can appear on the Wall and be manually cut to, but the scoring loop, photo capture, LiDAR, zoom, and camera-flip controls will not automatically target them unless the agent changes those filters.

This is probably desirable for the first pass:

Mac4 lanes: visible and manually cuttable on Wall.
iOS lanes: remain automatic director/scoring/photo/zoom/LiDAR targets.
Mac2 Arducam: remains evidence-style lane.

If the operator wants auto-cut to use Mac4 Camera 7/8/9, broaden `device_is_director_eligible` carefully and give the Mac4 lanes explicit `shot_role` and `priority_weight` values. Do not do this until frames are stable.

Suggested UI Labels

Use these labels in the Wall tiles:

7. MAC4 INSTA360 X4
8. MAC4 FEMTO MEGA
9. MAC4 FEMTO BOLT

Suggested model labels:

INSTA360 X4
ORBBEC FEMTO MEGA RGB
ORBBEC FEMTO BOLT RGB

Suggested evidence classification:

1. Keep only Mac2 Arducam as `EVIDENCE` for now, because Wall currently uses Arducam/record-agent heuristics for evidence styling.
2. Treat Mac4 Camera 8/9 as normal visual lanes unless the body-truth/reconstruction workflow explicitly wants them marked as evidence.

Test Plan For The Integrating Agent

1. Confirm Mac4 lane status:

curl -fsS 'http://[ip]:8087/status?[sensitive field redacted]
curl -fsS 'http://[ip]:8088/status?[sensitive field redacted]
curl -fsS 'http://[ip]:8089/status?[sensitive field redacted]

2. Confirm each lane can produce frames before marking `stream_ready: true`:

curl -m 8 -o /tmp/mac4-camera7.jpg 'http://[ip]:8087/snapshot?[sensitive field redacted]
curl -m 8 -o /tmp/mac4-camera8.jpg 'http://[ip]:8088/snapshot?[sensitive field redacted]
curl -m 8 -o /tmp/mac4-camera9.jpg 'http://[ip]:8089/snapshot?[sensitive field redacted]

3. Register the three lanes into `:9404`.
4. Confirm `/devices` includes all nine intended cameras:

curl -fsS 'http://[ip]:9404/devices' | python3 -m json.tool

5. Confirm the server fan-out sees frames:

curl -m 10 -o /tmp/server-mac4-camera7.jpg 'http://[ip]:9404/device/mac4-camera7-insta360/frame?mode=program'
curl -m 10 -o /tmp/server-mac4-camera8.jpg 'http://[ip]:9404/device/mac4-camera8-femto-mega/frame?mode=program'
curl -m 10 -o /tmp/server-mac4-camera9.jpg 'http://[ip]:9404/device/mac4-camera9-femto-bolt/frame?mode=program'

6. Open the Wall:

http://[ip]:9404/wall

7. Verify:

Header shows 9 cams when 9-up is active.
Tiles 7, 8, and 9 render labels correctly.
Existing six cameras still appear and can be cut.
Manual Cut works on Mac4 lanes.
Auto mode does not unexpectedly select Mac4 lanes unless intentionally enabled.
Record All still gates only cameras that can report recording results.

Known Non-Goals From This Pass

1. No direct Live Director or Wall code changes were made.
2. No existing Mac4 depth/body-truth process was stopped.
3. No automatic director scoring behavior was changed.
4. No Record All participation was added for Mac4 lanes.
5. No assumption was made that AVFoundation index ordering is stable.

Short Integration Summary

Mac4 now has three addressable HTTP camera lanes on ports `8087`, `8088`, and `8089`. The next agent should register those lanes into the existing MotionMix multicam server at `[ip]:9404`, then update `wall.html` to support a 9-camera picker/layout instead of the current six-device cap. The first safe milestone is status-visible and manually cuttable Mac4 tiles; full auto-director and recording support should wait until Mac4 frame delivery is stable.