cc-collection
High-performance sensor fusion library for motion capture data collection. Combines Sony Mocopi IMU sensors with MediaPipe pose estimation using Extended Kalman Filtering for ML training data generation.
Full Public Reader
cc-collection
High-performance sensor fusion library for motion capture data collection. Combines Sony Mocopi IMU sensors with MediaPipe pose estimation using Extended Kalman Filtering for ML training data generation.
Features
- Multi-Sensor Fusion: Combines Mocopi (6 IMU sensors) with MediaPipe (camera-based) tracking
- Extended Kalman Filter: 13D state estimation per limb with adaptive noise modeling
- Real-time Processing: 60Hz+ fusion with sub-millisecond latency
- Beat Synchronization: Phase-aligned motion vectors for music-driven applications
- Motion Transforms: 25D and 63D feature vectors for ML training
- Cross-Platform: Rust core with Python bindings (PyO3) and WebAssembly support
Architecture
┌─────────────────────────────────────────────────────────────────────┐
│ cc-collection │
├─────────────────────────────────────────────────────────────────────┤
│ Protocol Layer │ Fusion Layer │ Output Layer │
│ ┌──────────────────┐ │ ┌─────────────────┐ │ ┌────────────┐ │
│ │ MocopiParser │───┼─▶│ FusionEngine │──┼─▶│ To25D │ │
│ │ (UDP/JSON) │ │ │ (EKF per limb) │ │ │ Transform │ │
│ └──────────────────┘ │ └─────────────────┘ │ └────────────┘ │
│ ┌──────────────────┐ │ ┌─────────────────┐ │ ┌────────────┐ │
│ │ MediaPipeParser │───┼─▶│ TemporalAlign │──┼─▶│ To63D │ │
│ │ (JSON/WS) │ │ │ (Clock Sync) │ │ │ Transform │ │
│ └──────────────────┘ │ └─────────────────┘ │ └────────────┘ │
├─────────────────────────┴───────────────────────┴──────────────────┤
│ Bindings: Python (PyO3) │ WebAssembly │ C FFI │
└─────────────────────────────────────────────────────────────────────┘Installation
Rust
Add to your `Cargo.toml`:
[dependencies]
cc-collection = { path = "../core/cc-collection" }Python
# Install maturin
pip install maturin
# Build and install
cd core/cc-collection
maturin develop --features python
# Or install from wheel
maturin build --features python --release
pip install target/wheels/cc_collection-*.whlQuick Start
Rust
use cc_collection::{
FusionEngine, FusionConfig, To25DTransform,
MocopiParser, MediaPipeParser,
};
// Create fusion engine
let config = FusionConfig::default();
let mut engine = FusionEngine::new(config);
// Create parsers
let mut mocopi_parser = MocopiParser::new();
let mut mediapipe_parser = MediaPipeParser::new();
// Process frames
let mocopi_frame = mocopi_parser.parse(&udp_packet)?;
let mediapipe_frame = mediapipe_parser.parse_json(&json_data)?;
let skeleton = engine.process_frame(
Some(&mocopi_frame),
Some(&mediapipe_frame),
1.0 / 60.0, // dt
);
// Transform to motion vector
let mut transform = To25DTransform::new(60.0);
let motion_25d = transform.transform(&skeleton, beat_phase);Python
import cc_collection
# Create fusion engine
engine = cc_collection.FusionEngine(session_id="dance_001")
# Process frame
skeleton = engine.process(
mocopi_json=mocopi_data,
mediapipe_json=mediapipe_data,
dt=1/60
)
# Access limb data
hip_pos = skeleton.get_position("hip") # [x, y, z]
positions = skeleton.positions_array() # NumPy (14, 3)
quaternions = skeleton.quaternions_array() # NumPy (14, 4)
# Transform to motion vector
transform = cc_collection.To25DTransform(frame_rate=60.0)
motion = transform.transform(skeleton, beat_phase=0.5) # NumPy (25,)
# Validate quality
validator = cc_collection.MotionValidator()
report = validator.validate(skeleton)
print(f"Quality: {report.quality:.2f}, Valid: {report.is_valid}")API Overview
Core Types
| Type | Description |
|---|---|
| `LimbId` | Enum for 14 tracked limbs (Hip, Head, LeftShoulder, etc.) |
| `LimbState` | Per-limb state: position, quaternion, velocity, confidence |
| `FusedSkeleton` | Complete skeleton with all limb states |
| `FusionMode` | Current mode: None, MocopiOnly, MediaPipeOnly, Fused |
| `DataSource` | Source tracking: Mocopi, MediaPipe, Fused, Interpolated |
Fusion
| Component | Description |
|---|---|
| `FusionConfig` | Configuration: session_id, min_confidence, smoothing |
| `FusionEngine` | Main processor with EKF-based fusion |
| `FusionStats` | Runtime statistics: frame count, sync status |
Transforms
| Transform | Output | Description |
|---|---|---|
| `To25DTransform` | 25D vector | Core motion features for ML training |
| `To63DTransform` | 63D vector | Extended features with face/hands |
Protocol Parsers
| Parser | Input | Description |
|---|---|---|
| `MocopiParser` | UDP binary, JSON | Sony Mocopi protocol (27 bones) |
| `MediaPipeParser` | JSON | MediaPipe Holistic (pose, face, hands) |
Capture
| Component | Description |
|---|---|
| `CaptureSession` | Recording with beat sync and quality filtering |
| `CaptureRingBuffer` | Lock-free circular buffer for frames |
Motion Vector Format
25D Motion Vector
The 25D transform produces a compact representation optimized for ML training:
Index Description
───── ───────────────────────────────
0-2 Hip position (x, y, z)
3-5 Hip velocity (vx, vy, vz)
6-8 Head orientation (yaw, pitch, roll)
9-11 Left hand position relative to hip
12-14 Right hand position relative to hip
15-17 Left foot position relative to hip
18-20 Right foot position relative to hip
21-22 Arm extension (left, right) [0-1]
23 Overall confidence [0-1]
24 Beat phase [0-1]63D Extended Vector
Adds face expression and hand gesture features:
Index Description
───── ───────────────────────────────
0-24 Core 25D features
25-34 Face state (10 features)
35-48 Left hand (14 features)
49-62 Right hand (14 features)Limb Mapping
| Index | LimbId | Mocopi Sensor | MediaPipe Landmark |
|---|---|---|---|
| 0 | Hip | root (0) | 23-24 midpoint |
| 1 | Head | head (4) | 0 (nose) |
| 2 | LeftShoulder | l_shoulder (7) | 11 |
| 3 | RightShoulder | r_shoulder (8) | 12 |
| 4 | LeftElbow | l_elbow (9) | 13 |
| 5 | RightElbow | r_elbow (10) | 14 |
| 6 | LeftWrist | l_wrist (11) | 15 |
| 7 | RightWrist | r_wrist (12) | 16 |
| 8 | LeftHip | l_hip (13) | 23 |
| 9 | RightHip | r_hip (14) | 24 |
| 10 | LeftKnee | l_knee (15) | 25 |
| 11 | RightKnee | r_knee (16) | 26 |
| 12 | LeftAnkle | l_ankle (17) | 27 |
| 13 | RightAnkle | r_ankle (18) | 28 |
Performance
Benchmarked on Apple M1:
| Operation | Time | Throughput |
|---|---|---|
| Mocopi parse (binary) | 2.1 µs | 476k frames/sec |
| MediaPipe parse (JSON) | 8.3 µs | 120k frames/sec |
| Fusion (single frame) | 45 µs | 22k frames/sec |
| 25D transform | 3.2 µs | 312k frames/sec |
| Full pipeline | 58 µs | 17k frames/sec |
Configuration
FusionConfig
FusionConfig {
session_id: String, // Session identifier
min_confidence: f32, // Minimum confidence [0-1], default 0.3
enable_smoothing: bool, // Enable Kalman smoothing, default true
}Environment Variables
| Variable | Description | Default |
|---|---|---|
| `CC_FUSION_MIN_CONFIDENCE` | Minimum confidence threshold | 0.3 |
| `CC_FUSION_SMOOTHING` | Enable smoothing (0/1) | 1 |
| `CC_LOG_LEVEL` | Log level (trace/debug/info/warn/error) | info |
Testing
# Run all tests
cargo test
# Run with Python feature
cargo test --features python
# Run benchmarks
cargo bench
# Run specific test
cargo test fusion_engineExamples
Real-time Stream Processing
from cc_collection.compat import StreamProcessor
processor = StreamProcessor(
session_id="live_session",
min_confidence=0.6,
enable_smoothing=True,
)
while True:
result = processor.process_frame(
mocopi_json=get_mocopi_data(),
mediapipe_json=get_mediapipe_data(),
)
if result["is_valid"]:
motion_vector = result["motion_25d"]
# Send to ML model or visualizationTraining Data Collection
from cc_collection.compat import TrainingDataCollector
collector = TrainingDataCollector(
session_name="dance_training_001",
min_quality=0.7,
target_bpm=120.0,
)
collector.start_recording()
for frame in frame_source:
skeleton = engine.process(frame.mocopi, frame.mediapipe)
collector.record_frame(skeleton, frame.beat_phase, frame.quality)
collector.stop_recording()
# Export for training
data = collector.export_numpy()
np.savez("training_data.npz", **data)Batch Processing for ML
from cc_collection.compat import MotionDataLoader
loader = MotionDataLoader(
frame_rate=60.0,
normalize=True,
include_velocity=True,
)
for batch in loader.iter_batches(frames, batch_size=32):
motion_vectors = batch["motion_25d"] # (32, 25)
positions = batch["positions"] # (32, 14, 3)
# Train model...License
MIT License - see [LICENSE](LICENSE) for details.
Related Projects
- [cc-mcs](../../backend/cc-mcs) - Motion Capture Server with fusion API
- [cc-core](../cc-core) - Python ML training pipeline
- [cc-dashboard](../../apps/web/cc-dashboard) - Real-time visualization
Promotion Decision
Attach run IDs, datasets, metrics, and reproduction commands.
Source Anchor
Comp-Core/core/motion/cc-collection/README.md
Detected Structure
Method · Evaluation · Architecture