Grand Diomande Research · Full HTML Reader

Gesture Training System - Complete Guide

The **Gesture Training System** allows you to record, practice, and refine gesture patterns for high-accuracy recognition. Think of it like **training a muscle memory** - the more you practice, the better the system recognizes your unique gestures.

Agents That Account for Themselves proposal experiment writeup candidate score 36 .md

Full Public Reader

Gesture Training System - Complete Guide

Overview

The Gesture Training System allows you to record, practice, and refine gesture patterns for high-accuracy recognition. Think of it like training a muscle memory - the more you practice, the better the system recognizes your unique gestures.

---

System Components

### 1. Gesture Database (`gesture_database.py`)
- Stores trained gesture templates
- Manages training samples
- Computes statistical features (mean, std)

### 2. Gesture Recorder (`gesture_recorder.py`)
- Captures sensor + video data
- Extracts features from raw data
- Saves training samples

### 3. Gesture Recognizer (`gesture_recognizer.py`)
- Matches live gestures to templates
- Calculates similarity scores
- Provides feedback for improvement

### 4. Training UI (`training_ui.py`)
- Interactive interface for training
- Practice mode with real-time feedback
- Progress tracking and statistics

---

Quick Start

Step 1: Install Dependencies

bash

# Install required packages
pip install numpy websockets opencv-python google-genai

Step 2: Setup Phone

1. Install Sensor Logger app on your phone
2. Configure to stream to your computer's IP:
- Protocol: WebSocket
- Host: `your-computer-ip`
- Port: `8765`

Step 3: Run Training UI

bash

cd dj_agent/gesture_control/trainer
python training_ui.py

---

Training Workflow

Phase 1: Recording Samples (15+ per gesture)

┌─────────────────────────────────────────────────────────┐
│  Step 1: Select Gesture Name                            │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Enter gesture name: swipe_right                        │
│                                                         │
│  Current samples: 0                                     │
│  Target samples: 15                                     │
│  Remaining: 15                                          │
│                                                         │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│  Step 2: Record Sample                                  │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Prepare to record 'swipe_right'                        │
│  Press ENTER when ready...                              │
│                                                         │
│  3...                                                   │
│  2...                                                   │
│  1...                                                   │
│  🔴 GO!                                                  │
│                                                         │
│  [Recording for 2 seconds...]                           │
│                                                         │
│  ✅ Sample 1/15 recorded                                 │
│                                                         │
└─────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────┐
│  Step 3: Repeat Until Complete                          │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  🎉 Enough samples collected!                            │
│  Train template now? (y/n): y                           │
│                                                         │
│  ✅ Trained template for swipe_right (15 samples)        │
│                                                         │
└─────────────────────────────────────────────────────────┘

Phase 2: Practice Mode (Test & Improve)

┌─────────────────────────────────────────────────────────┐
│  PRACTICE RESULT                                        │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Overall Match: ████████████████████████████░░░░░░  85% │
│  Confidence:    ███████████████████████████████░░░  90% │
│                                                         │
│  Feature Breakdown:                                     │
│    ✓ accel_peak_x        ████████████████████  92%     │
│    ✓ accel_duration      ██████████████████░░  85%     │
│    ⚠ gyro_total_rotation ████████████░░░░░░░░  65%     │
│    ✓ gesture_duration    ███████████████████░  88%     │
│                                                         │
│  Feedback:                                              │
│    • Good horizontal speed                              │
│    • Timing slightly off                                │
│                                                         │
│  Suggestions:                                           │
│    • Add more rotation                                  │
│    • Make motion smoother                               │
│                                                         │
└─────────────────────────────────────────────────────────┘

Phase 3: Review & Refine

┌─────────────────────────────────────────────────────────┐
│  REVIEW MODE                                            │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  Per-Gesture Statistics:                                │
│                                                         │
│    swipe_right:                                         │
│      Samples: 15                                        │
│      Accuracy: 92%                                      │
│      Practice attempts: 8                               │
│      Average score: 87%                                 │
│      Trend: 📈 Improving!                                │
│                                                         │
│    circle_cw:                                           │
│      Samples: 18                                        │
│      Accuracy: 88%                                      │
│      Practice attempts: 12                              │
│      Average score: 85%                                 │
│      Trend: ➡️  Stable                                   │
│                                                         │
└─────────────────────────────────────────────────────────┘

---

Feature Extraction

Sensor Features (13 features)

#### Acceleration Features
1. `accel_peak_x` - Maximum X-axis acceleration (horizontal swipes)
2. `accel_peak_y` - Maximum Y-axis acceleration (vertical movements)
3. `accel_peak_z` - Maximum Z-axis acceleration (push/pull)
4. `accel_mean_magnitude` - Average total acceleration
5. `accel_duration` - Duration of significant motion

#### Gyroscope Features
6. `gyro_peak_x` - Maximum X-axis rotation
7. `gyro_peak_y` - Maximum Y-axis rotation
8. `gyro_peak_z` - Maximum Z-axis rotation (important for circles)
9. `gyro_total_rotation` - Total rotation angle
10. `gyro_direction` - Rotation direction ("cw", "ccw", or "none")

#### Temporal Features
11. `gesture_duration` - Total gesture duration
12. `num_peaks` - Number of distinct motion peaks
13. `motion_smoothness` - How smooth the motion is (0-1)

Feature Weights (Importance)

Feature	Weight	Importance
`gyro_direction`	2.0	Critical for CW/CCW circles
`accel_peak_x`	1.5	High for horizontal swipes
`accel_peak_y`	1.5	High for vertical movements
`gyro_total_rotation`	1.5	High for circles
`accel_duration`	1.3	Medium for timing
`gesture_duration`	1.3	Medium for timing
`accel_mean_magnitude`	1.2	Medium for intensity
`motion_smoothness`	0.8	Low (less critical)

---

Recognition Algorithm

Similarity Scoring

For each feature, compute normalized distance (z-score):

z_score = |value - template_mean| / template_std

Convert z-score to similarity (0-1):

similarity = exp(-z_score² / 2)

Interpretation:
- z = 0 → similarity = 1.0 (perfect match)
- z = 1 → similarity = 0.6 (within 1 standard deviation)
- z = 2 → similarity = 0.2 (within 2 standard deviations)
- z > 3 → similarity ≈ 0.0 (poor match)

Overall Match Score

match_score = Σ(similarity[feature] × weight[feature]) / Σ(weight)

Confidence Calculation

confidence = match_score × direction_penalty

Where `direction_penalty = 0.5` if rotation direction is wrong.

Recognition Threshold

Default: **70

---

Training Best Practices

1. Record 15+ Samples Per Gesture

Why: More samples = better statistical model

How:
- Record in batches (5 samples, then break)
- Vary your hand position slightly (don't be robotic)
- Maintain consistent core motion

2. Practice Until Consistent

Goal: Average practice score >85

Strategy:
- Practice 5-10 times after training
- Focus on features with low scores
- Read feedback carefully

3. Start with Simple Gestures

Easy gestures (train first):
- Swipe right/left (linear motion)
- Tap twice (discrete events)
- Tilt left/right (static position)

Hard gestures (train later):
- Circle CW/CCW (requires rotation precision)
- Figure-8 (complex trajectory)
- Multi-tap sequences (timing critical)

4. Use Consistent Grip

Recommendation: Hold phone the same way every time
- Portrait orientation
- Thumb on screen, fingers on back
- Firm but relaxed grip

5. Practice in Realistic Conditions

Good: Practice standing up (like live performance)
Bad: Practice sitting down (different body mechanics)

---

Troubleshooting

Issue: Low Recognition Accuracy (<80

Symptoms:

Feature Breakdown:
  ⚠ accel_peak_x        ██████░░░░░░░░░░░░░░  55%
  ⚠ accel_duration      ████░░░░░░░░░░░░░░░░  40%

Causes:
1. Inconsistent gesture execution
2. Insufficient training samples
3. Too much variation in samples

Solutions:
1. Re-record samples - Delete template and start fresh
2. Focus on one aspect - Practice just the speed, then direction, etc.
3. Add more samples - Record 20-30 instead of 15

Issue: Features Don't Match Template

Symptoms:

Suggestions:
  • Move phone faster
  • Add more rotation
  • Make motion smoother

Cause: Your live gesture differs from trained template

Solution:
1. Check what the template expects:

python

   template = db.get_template("swipe_right")
   print(f"Expected accel_peak_x: {template.sensor_mean.accel_peak_x:.2f}")

2. Adjust your motion to match template expectations

3. Or re-train if current template doesn't match your natural style

Issue: Direction Always Wrong

Symptoms:

Confidence: 50%  (penalized for wrong direction)
gyro_direction: "none" (expected: "cw")

Cause: Not adding enough rotation to gesture

Solution:
1. Exaggerate rotation - Make circles bigger and more circular
2. Check phone orientation - Make sure Z-axis is pointing up
3. Practice just rotation - Ignore other movements, focus on clean circles

---

Advanced Topics

Custom Feature Weights

If certain features are more important for your gestures:

python

recognizer.feature_weights['accel_peak_x'] = 3.0  # Make horizontal motion critical
recognizer.feature_weights['motion_smoothness'] = 0.5  # De-emphasize smoothness

Cross-Validation

Test template accuracy using held-out samples:

python

# In gesture_database.py (TODO: implement)
def cross_validate(gesture_name: str) -> float:
    """
    Test template accuracy using leave-one-out validation.

    Returns:
        Accuracy (0-1)
    """
    samples = self.get_samples(gesture_name)
    correct = 0

    for i in range(len(samples)):
        # Train on all except sample i
        train_samples = samples[:i] + samples[i+1:]
        template = self._train_from_samples(train_samples)

        # Test on sample i
        result = recognizer.compare_to_template(
            samples[i].sensor_features,
            template,
        )

        if result.confidence > 0.7:
            correct += 1

    return correct / len(samples)

Gesture Variants

Train multiple variants of same gesture:

swipe_right_fast   (duration < 0.3s)
swipe_right_normal (duration 0.3-0.5s)
swipe_right_slow   (duration > 0.5s)

Map all variants to same keyboard shortcut.

---

Database Structure

gesture_database/
├── templates.json          # All trained templates
├── samples/                # Training samples
│   ├── swipe_right_20250122_143022_123456.json
│   ├── swipe_right_20250122_143025_789012.json
│   ├── circle_cw_20250122_143030_345678.json
│   └── ...
└── config.json             # Database configuration

Template Format (JSON)

json

{
  "swipe_right": {
    "name": "swipe_right",
    "description": "Gesture: swipe_right",
    "keyboard_shortcut": "Cmd+Right",
    "rekordbox_action": "play_pause",
    "sensor_mean": {
      "accel_peak_x": 2.5,
      "accel_peak_y": 0.3,
      "accel_duration": 0.4,
      ...
    },
    "sensor_std": {
      "accel_peak_x": 0.3,
      "accel_peak_y": 0.1,
      ...
    },
    "num_samples": 15,
    "accuracy": 0.92,
    "created_at": "2025-01-22T14:30:00",
    "updated_at": "2025-01-22T14:35:00"
  }
}

---

Performance Metrics

Training Time

Task	Time	Notes
Record single sample	2s	Fixed duration
Extract features	<10ms	Fast computation
Train template	<100ms	Statistical aggregation
Total per gesture	~3 minutes	15 samples @ 2s each + breaks

Recognition Performance

Metric	Target	Typical
Accuracy (after training)	>90
Latency (sensor → match)	<50ms	20-30ms
False positive rate	<5
False negative rate	<10

---

Summary

Training Workflow

1. Record 15+ samples per gesture (3 min)
2. Train template from samples (<1 sec)
3. Practice until avg score >85
4. Refine based on feedback
5. Repeat for all gestures

Total Time Investment

Gestures	Training Time	Practice Time	Total
5 gestures	15 minutes	20 minutes	35 minutes
10 gestures	30 minutes	40 minutes	70 minutes
15 gestures	45 minutes	60 minutes	105 minutes

Expected Accuracy

After training: **>85
After practice: **>90
After 1 week of use: **>95

---

Master your gestures, master your performance! 🎯📱

Author: Computational Choreography
Version: 1.0 - Training System Guide

Promotion Decision

Attach run IDs, datasets, metrics, and reproduction commands.

Source Anchor

Comp-Core/apps/web/cc-studio/docs/dj_agent/gesture_control/training_system.md

Detected Structure

Method · Evaluation · Figures · Code Anchors · Architecture