Grand Diomande Research · Full HTML Reader

cc-collection

High-performance sensor fusion library for motion capture data collection. Combines Sony Mocopi IMU sensors with MediaPipe pose estimation using Extended Kalman Filtering for ML training data generation.

Embodied Trajectory Systems research note experiment writeup candidate score 18 .md

Full Public Reader

cc-collection

High-performance sensor fusion library for motion capture data collection. Combines Sony Mocopi IMU sensors with MediaPipe pose estimation using Extended Kalman Filtering for ML training data generation.

Features

  • Multi-Sensor Fusion: Combines Mocopi (6 IMU sensors) with MediaPipe (camera-based) tracking
  • Extended Kalman Filter: 13D state estimation per limb with adaptive noise modeling
  • Real-time Processing: 60Hz+ fusion with sub-millisecond latency
  • Beat Synchronization: Phase-aligned motion vectors for music-driven applications
  • Motion Transforms: 25D and 63D feature vectors for ML training
  • Cross-Platform: Rust core with Python bindings (PyO3) and WebAssembly support

Architecture

┌─────────────────────────────────────────────────────────────────────┐
│                         cc-collection                                │
├─────────────────────────────────────────────────────────────────────┤
│  Protocol Layer          │  Fusion Layer         │  Output Layer    │
│  ┌──────────────────┐   │  ┌─────────────────┐  │  ┌────────────┐  │
│  │  MocopiParser    │───┼─▶│  FusionEngine   │──┼─▶│ To25D      │  │
│  │  (UDP/JSON)      │   │  │  (EKF per limb) │  │  │ Transform  │  │
│  └──────────────────┘   │  └─────────────────┘  │  └────────────┘  │
│  ┌──────────────────┐   │  ┌─────────────────┐  │  ┌────────────┐  │
│  │  MediaPipeParser │───┼─▶│  TemporalAlign  │──┼─▶│ To63D      │  │
│  │  (JSON/WS)       │   │  │  (Clock Sync)   │  │  │ Transform  │  │
│  └──────────────────┘   │  └─────────────────┘  │  └────────────┘  │
├─────────────────────────┴───────────────────────┴──────────────────┤
│  Bindings: Python (PyO3) │ WebAssembly │ C FFI                      │
└─────────────────────────────────────────────────────────────────────┘

Installation

Rust

Add to your `Cargo.toml`:

toml
[dependencies]
cc-collection = { path = "../core/cc-collection" }

Python

bash
# Install maturin
pip install maturin

# Build and install
cd core/cc-collection
maturin develop --features python

# Or install from wheel
maturin build --features python --release
pip install target/wheels/cc_collection-*.whl

Quick Start

Rust

rust
use cc_collection::{
    FusionEngine, FusionConfig, To25DTransform,
    MocopiParser, MediaPipeParser,
};

// Create fusion engine
let config = FusionConfig::default();
let mut engine = FusionEngine::new(config);

// Create parsers
let mut mocopi_parser = MocopiParser::new();
let mut mediapipe_parser = MediaPipeParser::new();

// Process frames
let mocopi_frame = mocopi_parser.parse(&udp_packet)?;
let mediapipe_frame = mediapipe_parser.parse_json(&json_data)?;

let skeleton = engine.process_frame(
    Some(&mocopi_frame),
    Some(&mediapipe_frame),
    1.0 / 60.0,  // dt
);

// Transform to motion vector
let mut transform = To25DTransform::new(60.0);
let motion_25d = transform.transform(&skeleton, beat_phase);

Python

python
import cc_collection

# Create fusion engine
engine = cc_collection.FusionEngine(session_id="dance_001")

# Process frame
skeleton = engine.process(
    mocopi_json=mocopi_data,
    mediapipe_json=mediapipe_data,
    dt=1/60
)

# Access limb data
hip_pos = skeleton.get_position("hip")        # [x, y, z]
positions = skeleton.positions_array()         # NumPy (14, 3)
quaternions = skeleton.quaternions_array()     # NumPy (14, 4)

# Transform to motion vector
transform = cc_collection.To25DTransform(frame_rate=60.0)
motion = transform.transform(skeleton, beat_phase=0.5)  # NumPy (25,)

# Validate quality
validator = cc_collection.MotionValidator()
report = validator.validate(skeleton)
print(f"Quality: {report.quality:.2f}, Valid: {report.is_valid}")

API Overview

Core Types

TypeDescription
`LimbId`Enum for 14 tracked limbs (Hip, Head, LeftShoulder, etc.)
`LimbState`Per-limb state: position, quaternion, velocity, confidence
`FusedSkeleton`Complete skeleton with all limb states
`FusionMode`Current mode: None, MocopiOnly, MediaPipeOnly, Fused
`DataSource`Source tracking: Mocopi, MediaPipe, Fused, Interpolated

Fusion

ComponentDescription
`FusionConfig`Configuration: session_id, min_confidence, smoothing
`FusionEngine`Main processor with EKF-based fusion
`FusionStats`Runtime statistics: frame count, sync status

Transforms

TransformOutputDescription
`To25DTransform`25D vectorCore motion features for ML training
`To63DTransform`63D vectorExtended features with face/hands

Protocol Parsers

ParserInputDescription
`MocopiParser`UDP binary, JSONSony Mocopi protocol (27 bones)
`MediaPipeParser`JSONMediaPipe Holistic (pose, face, hands)

Capture

ComponentDescription
`CaptureSession`Recording with beat sync and quality filtering
`CaptureRingBuffer`Lock-free circular buffer for frames

Motion Vector Format

25D Motion Vector

The 25D transform produces a compact representation optimized for ML training:

Index   Description
─────   ───────────────────────────────
0-2     Hip position (x, y, z)
3-5     Hip velocity (vx, vy, vz)
6-8     Head orientation (yaw, pitch, roll)
9-11    Left hand position relative to hip
12-14   Right hand position relative to hip
15-17   Left foot position relative to hip
18-20   Right foot position relative to hip
21-22   Arm extension (left, right) [0-1]
23      Overall confidence [0-1]
24      Beat phase [0-1]

63D Extended Vector

Adds face expression and hand gesture features:

Index   Description
─────   ───────────────────────────────
0-24    Core 25D features
25-34   Face state (10 features)
35-48   Left hand (14 features)
49-62   Right hand (14 features)

Limb Mapping

IndexLimbIdMocopi SensorMediaPipe Landmark
0Hiproot (0)23-24 midpoint
1Headhead (4)0 (nose)
2LeftShoulderl_shoulder (7)11
3RightShoulderr_shoulder (8)12
4LeftElbowl_elbow (9)13
5RightElbowr_elbow (10)14
6LeftWristl_wrist (11)15
7RightWristr_wrist (12)16
8LeftHipl_hip (13)23
9RightHipr_hip (14)24
10LeftKneel_knee (15)25
11RightKneer_knee (16)26
12LeftAnklel_ankle (17)27
13RightAnkler_ankle (18)28

Performance

Benchmarked on Apple M1:

OperationTimeThroughput
Mocopi parse (binary)2.1 µs476k frames/sec
MediaPipe parse (JSON)8.3 µs120k frames/sec
Fusion (single frame)45 µs22k frames/sec
25D transform3.2 µs312k frames/sec
Full pipeline58 µs17k frames/sec

Configuration

FusionConfig

rust
FusionConfig {
    session_id: String,           // Session identifier
    min_confidence: f32,          // Minimum confidence [0-1], default 0.3
    enable_smoothing: bool,       // Enable Kalman smoothing, default true
}

Environment Variables

VariableDescriptionDefault
`CC_FUSION_MIN_CONFIDENCE`Minimum confidence threshold0.3
`CC_FUSION_SMOOTHING`Enable smoothing (0/1)1
`CC_LOG_LEVEL`Log level (trace/debug/info/warn/error)info

Testing

bash
# Run all tests
cargo test

# Run with Python feature
cargo test --features python

# Run benchmarks
cargo bench

# Run specific test
cargo test fusion_engine

Examples

Real-time Stream Processing

python
from cc_collection.compat import StreamProcessor

processor = StreamProcessor(
    session_id="live_session",
    min_confidence=0.6,
    enable_smoothing=True,
)

while True:
    result = processor.process_frame(
        mocopi_json=get_mocopi_data(),
        mediapipe_json=get_mediapipe_data(),
    )

    if result["is_valid"]:
        motion_vector = result["motion_25d"]
        # Send to ML model or visualization

Training Data Collection

python
from cc_collection.compat import TrainingDataCollector

collector = TrainingDataCollector(
    session_name="dance_training_001",
    min_quality=0.7,
    target_bpm=120.0,
)

collector.start_recording()

for frame in frame_source:
    skeleton = engine.process(frame.mocopi, frame.mediapipe)
    collector.record_frame(skeleton, frame.beat_phase, frame.quality)

collector.stop_recording()

# Export for training
data = collector.export_numpy()
np.savez("training_data.npz", **data)

Batch Processing for ML

python
from cc_collection.compat import MotionDataLoader

loader = MotionDataLoader(
    frame_rate=60.0,
    normalize=True,
    include_velocity=True,
)

for batch in loader.iter_batches(frames, batch_size=32):
    motion_vectors = batch["motion_25d"]  # (32, 25)
    positions = batch["positions"]         # (32, 14, 3)
    # Train model...

License

MIT License - see [LICENSE](LICENSE) for details.

Related Projects

  • [cc-mcs](../../backend/cc-mcs) - Motion Capture Server with fusion API
  • [cc-core](../cc-core) - Python ML training pipeline
  • [cc-dashboard](../../apps/web/cc-dashboard) - Real-time visualization

Promotion Decision

Attach run IDs, datasets, metrics, and reproduction commands.

Source Anchor

Comp-Core/core/motion/cc-collection/README.md

Detected Structure

Method · Evaluation · Architecture