Scripts Directory

Scripts Directory

This directory contains utility scripts, training scripts, demos, setup tools, and testing utilities for the CC-TPO project.

Structure

scripts/
├── training/           # ML training scripts
├── testing/            # Test scripts
├── demo/               # Demo and example scripts
├── setup/              # Setup and installation scripts
└── main.py             # Main project script

---

Training Scripts

Location: `scripts/training/`

### train_ircp_full_dataset.py
Trains the IRCP model on the full dataset (277 conversations).

Usage:

python scripts/training/train_ircp_full_dataset.py

Output: Model checkpoints in `training/ircp/full_dataset/`

### train_sentence_transformer_ircp.py
Trains the SentenceTransformer-based IRCP model.

Usage:

python scripts/training/train_sentence_transformer_ircp.py

### monitor_training.py
Monitors ongoing training progress and metrics.

Usage:

python scripts/training/monitor_training.py

### regenerate_embeddings_277.py
Regenerates embeddings for the 277-conversation dataset.

Usage:

python scripts/training/regenerate_embeddings_277.py

### warm_model.py
Warms up the model (preloading and caching).

Usage:

python scripts/training/warm_model.py

---

Testing Scripts

Location: `scripts/testing/`

### test_complete_ircp_implementation.py
Comprehensive test suite for the complete IRCP implementation.

Usage:

python scripts/testing/test_complete_ircp_implementation.py

Tests:
- Model loading
- Embedding generation
- Coordinate calculation
- Search functionality

### test_trained_ircp_model.py
Tests the trained IRCP model specifically.

Usage:

python scripts/testing/test_trained_ircp_model.py

Verifies:
- Model can be loaded from checkpoint
- Embeddings are consistent
- Performance metrics

---

Demo Scripts

Location: `scripts/demo/`

### create_search_demo.py
Creates an interactive search demonstration.

Usage:

python scripts/demo/create_search_demo.py

Output: Demo HTML report and results JSON

### real_world_examples.py
Run real-world examples showcasing IRCP capabilities.

Usage:

python scripts/demo/real_world_examples.py

Features:
- Example queries
- Visualization of results
- Ring topology demonstrations

---

Setup Scripts

Location: `scripts/setup/`

### install_global_tool.sh
Installs global development tools and dependencies.

Usage:

bash scripts/setup/install_global_tool.sh

Installs:
- Python dependencies
- Node.js packages
- System tools

### setup-liquid-chat.sh
Sets up the liquid chat application and backend.

Usage:

bash scripts/setup/setup-liquid-chat.sh

Actions:
- Installs dependencies for liquid-chat-ui
- Installs dependencies for liquid-chat-backend
- Creates necessary databases
- Configures environment variables

---

Main Script

### main.py
Main entry point for the project (if applicable).

Usage:

python scripts/main.py [options]

---

Prerequisites

Python Dependencies

pip install -r requirements-ircp.txt

### System Requirements
- Python 3.8+
- Node.js 18+ (for apps)
- SQLite 3

---

Common Workflows

1. Train a New IRCP Model

# Train the model
python scripts/training/train_ircp_full_dataset.py

# Test the trained model
python scripts/testing/test_trained_ircp_model.py

# Monitor training progress
python scripts/training/monitor_training.py

2. Run Demos

# Create search demo
python scripts/demo/create_search_demo.py

# Run real-world examples
python scripts/demo/real_world_examples.py

3. Setup Development Environment

# Install global tools
bash scripts/setup/install_global_tool.sh

# Setup liquid chat
bash scripts/setup/setup-liquid-chat.sh

4. Testing

# Run complete tests
python scripts/testing/test_complete_ircp_implementation.py

# Test trained model
python scripts/testing/test_trained_ircp_model.py

---

Environment Variables

Some scripts may require environment variables:

# For training
export IRCP_DATA_PATH="data/conversations.json"
export IRCP_MODEL_PATH="training/ircp/full_dataset/"

# For testing
export IRCP_TEST_DB="data/databases/conversations_fixed.db"

---

Output Locations

Training outputs: `training/ircp/outputs/`
Test results: `evaluation_results/`
Demo outputs: `docs/ircp/` (reports and visualizations)
Logs: `logs/`

---

Troubleshooting

### Import Errors
If you encounter import errors, ensure packages are in your Python path:

import sys
from pathlib import Path

PROJECT_ROOT = Path(__file__).parent.parent
sys.path.append(str(PROJECT_ROOT / "packages" / "ircp"))
sys.path.append(str(PROJECT_ROOT / "packages" / "tpo"))

### Database Not Found
Ensure databases are in the correct location:
- `data/databases/conversations_fixed.db`
- `data/databases/claude_full_embeddings_dlm_fixed.db`

### Model Not Found
Check that the model is in the training directory:
- `training/ircp/full_dataset/best_model.pt`
- `training/ircp/full_dataset/inferred_config.json`

---

Contributing

When adding new scripts:

1. Place in the appropriate subdirectory (training, testing, demo, setup)
2. Add a docstring explaining the script's purpose
3. Update this README with usage instructions
4. Ensure the script uses relative paths to the project root