Research: Depth-Reactive Interactive Visual System in Unity 6 on macOS Apple Silicon

Research: Depth-Reactive Interactive Visual System in Unity 6 on macOS Apple Silicon

Date: 2026-04-03
Scope: Hardware selection, SDK viability, rendering pipeline, fluid simulation, and reference implementations for building a depth-camera-driven interactive particle/fluid installation using Unity 6 on M-series Macs.

---

Executive Summary

Building a depth-reactive visual system in Unity 6 on macOS Apple Silicon is feasible but requires navigating a fragmented SDK landscape. The Orbbec Femto Bolt is the strongest hardware choice (identical ToF sensor to the discontinued Azure Kinect, macOS supported via SDK v2, $299). However, the official Unity wrapper is stale (v1.1.9, Nov 2023) and does not yet wrap SDK v2. The recommended workaround is the K4A Wrapper path or the LightBuzz third-party SDK. Unity VFX Graph works on Metal and supports millions of particles on M-series GPUs, though the Unity 6 editor still requires Rosetta. For fluid simulation, Keijiro Takahashi's StableFluids (updated for Unity 6) is the canonical starting point, and his Rsvfx project is the exact reference architecture for depth-camera-to-VFX-Graph point cloud rendering. Apple LiDAR via Record3D is a viable alternative/supplementary depth source with a proven Unity VFX Graph pipeline.

---

1. Orbbec Femto Bolt SDK for Unity

### Hardware Specs
- Depth sensor: Microsoft 1MP iToF (identical to Azure Kinect DK)
- Depth modes: NFOV 640x576 @ 30fps (75x65 deg), WFOV 1024x1024 @ 15fps (120x120 deg), WFOV binned 512x512 @ 30fps
- RGB: 3840x2160 @ 30fps, HDR capable (81.1dB dynamic range, vs Azure Kinect's 45.6dB)
- IMU: 6-axis (accelerometer + gyroscope)
- Size: 115.3 x 40.3 x 65.0mm, 348g (smaller/lighter than Azure Kinect)
- Price: ~$299 USD
- Connection: USB-C (power + data)

SDK Status

Orbbec SDK v2 (recommended)
- Open-source, actively maintained (2,513 commits on main branch)
- macOS support: confirmed for "M2 chip, OS version 13.2" (ARM64 native)
- Supports Femto Bolt with full maintenance
- C/C++ API, Python wrapper, ROS1/2, K4A Wrapper
- NO Unity wrapper for SDK v2 yet -- this is the critical gap

Orbbec Unity SDK (v1-based)
- Version: v1.1.9 (last updated November 23, 2023)
- Ships with OrbbecSDK 1.10.9 (latest SDK v1 is 1.10.27)
- macOS NOT listed as supported in the Unity wrapper README (Windows + Android only)
- 18 cameras supported including Femto Bolt
- 10 sample scenes: HelloOrbbec, ColorViewer, DepthViewer, IRViewer, SyncAlignViewer, SensorControl, RecordPlayback, IMUReader, Pointcloud, DoubleIRViewer
- GitHub: https://github.com/orbbec/OrbbecUnitySDK

K4A Wrapper (best workaround)
- Reimplements Azure Kinect Sensor SDK API using Orbbec SDK v2 under the hood
- macOS 11+ supported, M1/M2 compatible
- Allows existing Azure Kinect Unity code to work with Femto Bolt
- Community-recommended path per Orbbec forum discussions
- GitHub: https://github.com/orbbec/OrbbecSDK-K4A-Wrapper

### Known Issues on macOS
- Root privileges required to connect to Orbbec cameras on macOS (confirmed in TouchDesigner docs, likely applies to Unity)
- Body tracking does NOT work on macOS -- Microsoft Body Tracking SDK is Windows-only. Orbbec recommends LightBuzz as alternative
- SDK v2 Unity wrapper has no official timeline -- community is asking, Orbbec has not responded with dates

### Recommended Integration Path
1. Use Orbbec SDK v2 C API directly via P/Invoke from C# in Unity, OR
2. Use K4A Wrapper + rfilkov's "Azure Kinect and Femto Bolt Examples for Unity" (Unity Asset Store, $25-50, 35+ demo scenes) -- Windows only for body tracking, OR
3. Use LightBuzz Body Tracking SDK ($25.76 on Asset Store) for cross-platform body tracking on macOS with Orbbec cameras
4. For point-cloud-only (no skeleton), write a thin C# wrapper around SDK v2's C API -- the depth/color streams are straightforward

### Sources
- [Orbbec Femto Bolt Product Page](https://www.orbbec.com/products/tof-camera/femto-bolt/)
- [OrbbecUnitySDK GitHub](https://github.com/orbbec/OrbbecUnitySDK)
- [OrbbecSDK v2 GitHub](https://github.com/orbbec/OrbbecSDK_v2)
- [K4A Wrapper GitHub](https://github.com/orbbec/OrbbecSDK-K4A-Wrapper)
- [Orbbec Community: SDK v2 Unity Integration](https://3dclub.orbbec3d.com/t/orbbecsdk-v2-unity-integration/4556)
- [Orbbec Community: macOS Support](https://3dclub.orbbec3d.com/t/mac-os-support-orbbec-sdk-1-9-3-and-later/3986)
- [Tegakari: Femto Bolt vs Azure Kinect Technical Comparison](https://www.tegakari.net/en/2024/09/techinfo-orbbec-femto-bolt/)

---

2. Unity VFX Graph on Metal (macOS)

### Current Capabilities
- VFX Graph is GPU-compute-based, runs entirely on compute shaders
- Metal is a supported backend -- VFX Graph works on macOS via Metal API
- Unity 6 VFX Graph requires HDRP or URP (not Built-in Render Pipeline)
- Supports Shader Graph integration for custom particle rendering
- Node-based authoring with GPU simulation

### Unity 6 on Apple Silicon Status
- Unity 6 editor still requires Rosetta as of 2026. The editor is NOT fully Apple Silicon native
- Rosetta must be installed for external processes (licensing, some build tools)
- Runtime builds can target Apple Silicon natively
- Apple announced Metal 4 at WWDC 2025 with neural rendering, MetalFX Frame Interpolation -- Unity expected to add plugin support

### Particle Count Performance (estimated from community reports)
- M1 Max: Users report smooth operation with VFX Graph, no specific particle ceiling published
- General VFX Graph: "Millions possible" per Unity documentation when properly optimized
- Key optimization: Keep "Initialize Particle" capacity as low as possible -- oversized capacity degrades performance even at low spawn rates
- Metal limitations: No atomic operations on textures (iOS/tvOS), no GetDimensions on buffers. Some ComputeBuffer binding issues reported
- Practical ceiling on M-series: Expect 500K-2M particles at 60fps for simple particles (position + color + size), significantly less for complex shaders with lighting/fluid coupling
- dilmerv/UnityVFXMillionsOfParticles: Reference repo with 1M+ particle effects on VFX Graph

### Known Metal Issues
- "Metal: Vertex or Fragment Shader requires a ComputeBuffer at index 5 to be bound" -- reported when creating VFX Graph assets on Metal (Unity Issue Tracker)
- Some Shader Graph + VFX Graph combinations can fail silently on Metal
- Workaround: Test on Metal frequently during development, not just at the end

### Sources
- [Unity Discussions: VFX Graph on Apple Silicon M1](https://discussions.unity.com/t/vfx-graph-and-apple-silicon-m1/897598)
- [Unity 6 macOS Requirements](https://docs.unity3d.com/6000.3/Documentation/Manual/macos-requirements-and-compatibility.html)
- [Unity Issue Tracker: Metal VFX Graph Error](https://issuetracker.unity3d.com/issues/metal-vfx-graph-creating-a-vfx-graph-asset-throws-an-error)
- [Unity Discussions: 2026 Apple Silicon Support](https://discussions.unity.com/t/lets-celebrate-together-2026-and-still-no-fully-native-apple-silicon-support/1696717)
- [Getting Started with VFX Graph in Unity 6](https://discussions.unity.com/t/getting-started-with-the-vfx-graph-in-unity-6/1617550)
- [Unity 6 VFX Graph E-Book](https://unity.com/blog/unity-6-vfx-graph-ebook)
- [dilmerv/UnityVFXMillionsOfParticles](https://github.com/dilmerv/UnityVFXMillionsOfParticles)

---

3. GPU Fluid Simulation in Unity

### Approach 1: Keijiro's StableFluids (RECOMMENDED)
- Targets Unity 6 (recently updated)
- GPU implementation of Jos Stam's "Stable Fluids" algorithm
- Exposes `FluidSimulation.VelocityField` for visualization and external force injection
- Codebase: 44.6
- Works with any render pipeline (URP/HDRP compatible)
- Lightweight, well-architected, easy to extend
- GitHub: https://github.com/keijiro/StableFluids

### Approach 2: IRCSS Compute-Shaders-Fluid-Dynamic
- Full fluid simulation pipeline in compute shaders
- 3 demo scenes: 2DFluid, 2DFluid Arbitrary Boundary, Persian Garden (fake 3D mapped to plane)
- FluidSimulator class with composable solver steps (advection, diffusion, projection)
- Handles velocity, dye, temperature fields
- Limitation: Tested on Unity 2019.1.14f1 + Windows DX11 + Legacy renderer only
- Author (Shahriar Shahrabi) wrote excellent accompanying blog posts on the theory
- GitHub: https://github.com/IRCSS/Compute-Shaders-Fluid-Dynamic-

### Approach 3: Fluid Frenzy (Commercial)
- GPU-accelerated 2.5D fluid simulation (shallow water equations)
- v1.3.2 (Nov 2025) -- actively maintained
- Designed for terrain interaction, not free-form particle fluids
- GPU quadtree LOD system
- Steam, particle effects, lava/water interaction
- Limitation: 2.5D only (one height per pixel), oriented toward terrain games not abstract installations
- Unity Asset Store: https://assetstore.unity.com/packages/vfx/shaders/fluid-frenzy-273366

### Approach 4: SPH (Research-Grade)
- 2025 IEEE paper: GPU-accelerated SPH in Unity3D with novel Count Sort + Parallel Prefix Scan for spatial hashing
- O(n) neighborhood search -- significant improvement for large-scale particle-based fluids
- Research code, not production-ready package

### Approach 5: LBM (Lattice Boltzmann)
- mattatz/unity-lbm-fluid-simulation: Compute shader implementation
- Good for 2D fluids with complex boundaries
- GitHub: https://github.com/mattatz/unity-lbm-fluid-simulation

### Recommendation for Depth-Reactive Installation
Use Keijiro's StableFluids as the fluid engine (it targets Unity 6, is clean code, and exposes the velocity field for injection). Map depth camera silhouette data into the velocity field as external forces. Render the fluid field as a VFX Graph texture input, driving particle behavior. This separates fluid simulation (compute shader) from particle rendering (VFX Graph).

### Sources
- [keijiro/StableFluids](https://github.com/keijiro/StableFluids)
- [IRCSS/Compute-Shaders-Fluid-Dynamic-](https://github.com/IRCSS/Compute-Shaders-Fluid-Dynamic-)
- [Gentle Introduction to Realtime Fluid Simulation](https://shahriyarshahrabi.medium.com/gentle-introduction-to-fluid-simulation-for-programmers-and-technical-artists-7c0045c40bac)
- [Fluid Frenzy Docs](https://frenzybyte.github.io/fluidfrenzy/docs/index/)
- [mattatz/unity-lbm-fluid-simulation](https://github.com/mattatz/unity-lbm-fluid-simulation)

---

4. Unity + Depth Camera Point Cloud Rendering

### Reference Architecture: Keijiro's Rsvfx (THE template)
- Connects Intel RealSense depth camera to Unity VFX Graph
- PointCloudBaker component: converts point cloud stream into two animated attribute maps:
- Position map (RGBAFloat texture where RGB = world XYZ)
- Color map (RGBA texture from RGB camera)
- These feed into VFX Graph's "Set Position/Color from Map" blocks
- Same pipeline as point cache files -- standard VFX Graph workflow
- Targets Unity 2019.2+ (needs updating for Unity 6 but architecture is sound)
- 100
- Fork exists for Kinect: IxxyXR/Rsvfx-Kinect adds Kinect support to the same architecture
- GitHub: https://github.com/keijiro/Rsvfx

### Adapting Rsvfx Pattern for Femto Bolt
The Rsvfx architecture is sensor-agnostic at the VFX Graph level. Replace the RealSense data source with Femto Bolt:
1. Read depth + color frames from Orbbec SDK v2 (C# P/Invoke or K4A Wrapper)
2. Project depth pixels to 3D points using camera intrinsics
3. Write to position map (RenderTexture, RGBAFloat) and color map (RenderTexture, RGBA8)
4. Feed textures to VFX Graph via "Set Position from Map" + "Set Color from Map" blocks
5. VFX Graph handles all particle spawning, simulation, rendering

### Other Point Cloud Rendering Options
- FastPoints: State-of-the-art point cloud renderer for Unity, handles 100M+ points, academic paper (arXiv:2302.05002) -- better for static/imported clouds than real-time streaming
- Pcx (keijiro): Point cloud importer for PLY files, VFX Graph compatible
- PcxEffects3 (keijiro): VFX Graph effects driven by point cloud data
- Unity Asset Store: VFX PointCloud Loader PLY: Commercial option for PLY files

### Key Insight
For real-time depth-reactive visuals, you do NOT render the point cloud directly. Instead, you use the depth data as an attribute map driving particle behavior in VFX Graph. The particles can be styled as anything -- fluid, sparks, geometry, volumetric fog. The point cloud is the input signal, not the visual output.

### Sources
- [keijiro/Rsvfx](https://github.com/keijiro/Rsvfx)
- [IxxyXR/Rsvfx-Kinect](https://github.com/IxxyXR/Rsvfx-Kinect)
- [keijiro/Pcx](https://github.com/keijiro/Pcx)
- [keijiro/PcxEffects3](https://github.com/keijiro/PcxEffects3)
- [FastPoints](https://github.com/eliasnd/FastPoints)
- [Depthkit VFX Graph Documentation](https://docs.depthkit.tv/docs/visual-effect-graph)

---

5. Orbbec Femto Bolt vs Azure Kinect DK

### Verdict
Femto Bolt is the unambiguous successor. It uses the identical depth sensor, has better RGB calibration, adds HDR, is smaller/lighter, is actively supported, works on macOS, and costs less than Azure Kinect's secondary market price. The only downside is the stale Unity wrapper, which is mitigated by the K4A compatibility layer.

### Sources
- [Orbbec: Femto Bolt vs Azure Kinect DK](https://www.orbbec.com/documentation/comparison-with-azure-kinect-dk/)
- [Tegakari Technical Comparison](https://www.tegakari.net/en/2024/09/techinfo-orbbec-femto-bolt/)
- [Depthkit: Femto Bolt for Volumetric Video](https://www.depthkit.tv/posts/new-depth-camera-support-orbbec-femto-bolt)
- [JetsonHacks: Femto Bolt as Azure Kinect Replacement](https://jetsonhacks.com/2024/07/07/orbbec-femto-bolt-a-microsoft-azure-kinect-replacement/)

---

6. Intel RealSense Viability in 2026

### Status: DEAD for new projects
- Intel announced discontinuation of the RealSense product lineup
- LiDAR and Tracking product families officially discontinued
- Stereo products (D400 series) technically still available but receiving no SDK updates
- macOS: Officially supports only macOS High Sierra (2017). Community builds exist for Monterey (Intel + Apple Silicon via LightBuzz guide) but are unsupported
- SDK (librealsense) is community-maintained on GitHub but no Intel investment
- pyrealsense2-macosx exists on PyPI but is a third-party effort

### What Still Works
- D435/D455 stereo cameras still function with librealsense
- keijiro's Rsvfx was built for RealSense and demonstrates the architecture well
- Community macOS builds may work but are fragile and unsupported

### Recommendation
Do not choose RealSense for a new project in 2026. Use it only if you already own the hardware and are on Windows/Linux. For macOS Apple Silicon, there is no reliable path forward.

### Sources
- [Intel RealSense Discontinuation FAQ](https://support.intelrealsense.com/hc/en-us/articles/10705277802643)
- [LightBuzz: Build RealSense for macOS Monterey](https://lightbuzz.com/realsense-macos/)
- [librealsense GitHub: Issue #9648 - Long Term Support](https://github.com/IntelRealSense/librealsense/issues/9648)
- [macOS Installation Docs](https://dev.intelrealsense.com/docs/macos-installation-for-intel-realsense-sdk)

---

7. Apple LiDAR as Depth Source

### What It Provides
- Available on: iPhone 12 Pro+, iPad Pro (2020+), Apple Vision Pro
- Range: ~5m indoor
- Resolution: 256x192 depth map at 60fps (sceneDepth API)
- Also provides scene reconstruction mesh via LiDAR scanner
- Body tracking: ARKit detects human joints (3D body pose) on LiDAR devices
- Semantic segmentation of depth data available

### Streaming to Unity on Mac: Record3D (PROVEN)
- Record3D iOS app ($3.99) streams RGBD data over USB to Mac/PC
- C++ bridging library `record3d_unity_streaming` converts to attribute maps
- Directly feeds Unity VFX Graph via position map + color map (same pattern as Rsvfx)
- Supports both TrueDepth (FaceID) and LiDAR cameras
- macOS support: yes, via `librecord3d_unity_streaming.dylib` in /usr/local/lib/
- GitHub demo: https://github.com/marek-simonik/record3d_unity_demo
- Last updated: v1.4 (May 2024)

### Limitations vs Dedicated Depth Camera
- 256x192 depth resolution is 6x lower than Femto Bolt WFOV (1024x1024)
- Requires iPhone/iPad tethered via USB (wireless possible but higher latency)
- No wide field of view option (fixed ~60 deg)
- Single device limits installation flexibility (can't easily ceiling-mount an iPhone)
- Phone heating under sustained LiDAR streaming

### Use Cases Where Apple LiDAR Excels
- Rapid prototyping (use the phone you already have)
- Portable demo installations
- Supplementary body tracking alongside a primary depth camera
- AR overlay development where the phone IS the display

### Sources
- [Record3D](https://record3d.app/)
- [record3d_unity_demo](https://github.com/marek-simonik/record3d_unity_demo)
- [record3d_unity_streaming](https://github.com/marek-simonik/record3d_unity_streaming)
- [ARKit Depth Data](https://developer.apple.com/documentation/arkit/ardepthdata)
- [LightBuzz: Body Tracking with ARKit LiDAR](https://lightbuzz.com/body-tracking-arkit-lidar/)

---

8. Duncan Fewkes / Holovis

### Search Results
No specific results found for "Duncan Fewkes" in connection with Holovis, interactive depth installations, or Unity VFX Graph work. Multiple search variations were attempted:
- "Duncan Fewkes artist interactive installation depth sensor"
- "Duncan Fewkes Holovis interactive depth Unity VFX Graph"
- "Holovis Duncan Fewkes developer engineer Unity"

None returned results tied to this individual. This person may operate under a different public name, or their work may be shared primarily on Instagram/social media without indexed web presence.

### Holovis (the company)
- What they are: Global leader in complex AV, immersive media, and interactive installations for theme parks, entertainment centers, museums
- Key tech: HoloTrac (computer vision and tracking for personalized nonlinear experiences)
- Notable project: Squid Game: The Experience (Bluetooth, accelerometer motion sensing, concealed sensors for real-time interactivity)
- They use Unity: Job postings for "3D GUI Developer - C# and Unity" confirmed
- Approach: Industrial-scale installations with concealed sensors, typically not depth-camera-only but multi-modal sensing
- Website: https://www.holovis.com/

### Closest Reference Work (Alternative)
If the Instagram content shows depth-reactive particle effects, the closest documented technical reference is Keijiro Takahashi's body of work:
- Rsvfx (depth camera to VFX Graph)
- Dkvfx (Depthkit footage in VFX Graph)
- StableFluids (GPU fluid simulation)
- VfxGraphTestbed (general VFX Graph experiments)
- PcxEffects3 (point cloud VFX)
- Depthkit Creator Profile: https://www.depthkit.tv/posts/keijiro-takahashi-creator-profile-depthkit

### Sources
- [Holovis Website](https://www.holovis.com/)
- [Holovis Capabilities](https://www.holovis.com/capabilities)
- [keijiro GitHub](https://github.com/keijiro)
- [Depthkit: Keijiro Creator Profile](https://www.depthkit.tv/posts/keijiro-takahashi-creator-profile-depthkit)

---

Architecture Recommendation

### Hardware
Primary: Orbbec Femto Bolt ($299)
- WFOV 1024x1024 @ 15fps for maximum coverage, or NFOV 640x576 @ 30fps for smoother interaction
- USB-C to Mac

Prototyping/Supplement: iPhone/iPad Pro with Record3D
- Quick iteration before Femto Bolt arrives
- Validates the full pipeline with hardware you already own

Software Stack

[Femto Bolt] --USB-C--> [Orbbec SDK v2 C API / P/Invoke]
                              |
                    [PointCloudBaker.cs]
                    (Rsvfx pattern: depth -> position map,
                     color -> color map, both RenderTextures)
                              |
                    [StableFluids compute shader]
                    (depth silhouette -> velocity field forces)
                              |
                    [VFX Graph]
                    (Set Position from Map + fluid velocity input)
                    (Set Color from Map or procedural color)
                              |
                    [HDRP or URP Renderer]
                    (Metal backend on macOS)
                              |
                    [Output: Projector / LED Wall / Display]

Phase Plan

Phase 1: Prototype with Record3D (1-2 days)
- Clone record3d_unity_demo
- Get LiDAR point cloud streaming into VFX Graph on Mac
- Validate Metal rendering pipeline
- Test particle counts and frame rates on your M-series chip

Phase 2: Integrate StableFluids (2-3 days)
- Clone keijiro/StableFluids
- Inject depth silhouette data as forces into the velocity field
- Feed velocity field to VFX Graph particles as external force
- Tune interaction feel (force magnitude, dissipation, viscosity)

Phase 3: Switch to Femto Bolt (1-2 days)
- Install Orbbec SDK v2 for macOS
- Write C# wrapper using P/Invoke (or use K4A Wrapper if the rfilkov Unity asset is acceptable)
- Replace Record3D data source with Femto Bolt depth stream
- Benefit: 4-16x higher depth resolution, wider FOV, no phone required

Phase 4: Polish and Scale (ongoing)
- Optimize particle counts for target frame rate
- Add visual layers (volumetric fog, trails, secondary particle systems)
- Calibrate depth camera to projection surface
- Multi-person interaction handling
- Lighting/color design for installation space

Key Risks

---

Body Tracking Options on macOS (if needed)

---

Summary of Recommendations

1. Buy the Femto Bolt -- it is the clear winner. Identical sensor to Azure Kinect, actively supported, macOS compatible at the SDK level, $299.

2. Start with Record3D + iPhone -- validate the full pipeline (depth -> attribute maps -> VFX Graph -> fluid interaction) in 1-2 days with hardware you own.

3. Use Keijiro's projects as your foundation -- Rsvfx (point cloud to VFX Graph architecture), StableFluids (fluid simulation targeting Unity 6). These are the canonical references.

4. Skip RealSense -- dead product line, no macOS future.

5. Write a thin C# SDK v2 bridge for the Femto Bolt in Unity rather than waiting for the official Unity wrapper (no timeline given by Orbbec). The C API is clean and the PointCloudBaker pattern from Rsvfx tells you exactly what data you need.

6. VFX Graph on Metal works -- expect 500K-2M simple particles at 60fps on M-series. The Unity 6 editor requires Rosetta but this is a development inconvenience, not a blocker.

7. If you need body tracking on macOS, use LightBuzz SDK. If you only need depth silhouette interaction (no skeleton joints), skip body tracking entirely and work directly with the depth map as a force field.