DLM Package Refactoring Audit
**Date:** 2025-12-08 **Status:** 🔍 In Progress **Purpose:** Comprehensive audit and refactoring plan to reduce technical debt
Full Public Reader
DLM Package Refactoring Audit
Date: 2025-12-08
Status: 🔍 In Progress
Purpose: Comprehensive audit and refactoring plan to reduce technical debt
---
Executive Summary
The DLM package has grown to 63,339 lines across 154 Python files with significant technical debt and consolidation opportunities.
Key Findings
1. Large Files: 10 files exceed 1,000 lines (should be <500)
2. Duplicate Functionality: Multiple embedders, loaders, and config systems
3. Unclear Organization: Mixed concerns (response/vangaurd, engine overlap)
4. Legacy Code: Old implementations (legacy_utils.py, deprecated embedders)
5. Test Fragmentation: Tests scattered across multiple locations
Critical Issues
| Issue | Severity | Files Affected | Impact |
|---|---|---|---|
| Pydantic v2 Incompatibility | 🚨 Critical | models/*.py, base.py | Blocks imports |
| Duplicate Embedders | ❌ High | engine/embedder.py (1604 lines), engine/ircp_embedder.py | Confusion |
| Mega Files | ⚠️ High | inference/artificial.py (3691 lines) | Unmaintainable |
| Legacy Utils | ⚠️ Medium | utils/legacy_utils.py (1518 lines) | Tech debt |
| Test Scatter | ⚠️ Medium | tests/, core/tests/ | Fragmentation |
---
Current Structure Analysis
Directory Overview
dlm/
├── core/ # Week 2-3 NEW (unified, clean)
│ ├── coordinates.py 827 lines ✅
│ ├── embeddings.py 295 lines ✅
│ ├── data_loader.py 576 lines ✅
│ ├── adapters.py 296 lines ✅
│ └── ircp/ Reference wrappers ✅
│
├── evaluation/ # Week 3 NEW (clean)
│ ├── metrics.py 447 lines ✅
│ └── validators.py 375 lines ✅
│
├── explainability/ # Week 3 NEW (clean)
│ ├── analyzer.py 595 lines ✅
│ ├── debugger.py 450 lines ✅
│ └── visualizer.py 350 lines ✅
│
├── pipeline/ # Week 3 NEW (clean)
│ ├── checkpoint_manager.py 370 lines ✅
│ ├── data_pipeline.py 330 lines ✅
│ └── training_pipeline.py 480 lines ✅
│
├── engine/ # LEGACY (overlaps with core)
│ ├── embedder.py 1604 lines ❌ CONSOLIDATE
│ ├── ircp_embedder.py 300 lines ❌ DEPRECATED
│ ├── builder.py 1004 lines ⚠️ REVIEW
│ ├── structure.py 851 lines ⚠️ REVIEW
│ ├── loader.py 564 lines ❌ DUPLICATE (core/data_loader)
│ └── ... (11 more files)
│
├── inference/ # ACTIVE (but needs cleanup)
│ ├── artificial.py 3691 lines 🚨 TOO LARGE
│ ├── prompt.py 2152 lines 🚨 TOO LARGE
│ ├── generator.py 1432 lines ⚠️ LARGE
│ ├── cloud_manager.py 834 lines ⚠️ REVIEW
│ └── ... (9 more files)
│
├── relationship/ # ACTIVE (complex but necessary)
│ ├── graph.py 1639 lines ⚠️ LARGE
│ ├── merger.py 1146 lines ⚠️ LARGE
│ ├── estimator.py 1064 lines ⚠️ LARGE
│ └── base.py 999 lines ⚠️ LARGE
│
├── response/ # ACTIVE (user-facing techniques)
│ ├── links.py 2083 lines 🚨 TOO LARGE
│ ├── system.py 1520 lines 🚨 TOO LARGE
│ ├── vangaurd/ 30+ files ⚠️ MANY FILES
│ │ ├── motion.py 1782 lines 🚨 TOO LARGE
│ │ ├── barista.py 670 lines
│ │ ├── synth/ 15+ files ⚠️ ORGANIZE
│ │ └── ... (many more)
│ └── ... (13 more files)
│
├── models/ # LEGACY (Pydantic v1)
│ ├── All files ~2000 lines 🚨 PYDANTIC V2 ISSUE
│ └── ... (9 files)
│
├── transformation/ # ACTIVE (coordinate transforms)
│ ├── tree.py 695 lines
│ ├── hypnotherapy.py 557 lines
│ └── ... (7 files)
│
├── services/ # UTILITY (metrics, preprocessing)
│ ├── metrics.py 1389 lines ⚠️ LARGE
│ └── ... (5 files)
│
├── callbacks/ # ACTIVE (streaming, events)
│ ├── manager.py 770 lines
│ ├── streaming.py 654 lines
│ └── base.py 579 lines
│
├── utils/ # MIXED
│ ├── legacy_utils.py 1518 lines ❌ LEGACY
│ └── logger.py New logger ✅
│
└── tests/ # FRAGMENTED
├── tests/ Main test dir ✅
└── core/tests/ Duplicate location ⚠️---
Consolidation Opportunities
1. Embedders (CRITICAL)
Current State:
- `engine/embedder.py` (1604 lines) - Old implementation
- `engine/ircp_embedder.py` (300 lines) - Deprecated
- `core/embeddings.py` (295 lines) - NEW Week 2 implementation
Action:
- ✅ KEEP: `core/embeddings.py` (already clean)
- ❌ DEPRECATE: `engine/embedder.py` → Move to `_deprecated/`
- ❌ DELETE: `engine/ircp_embedder.py` (superseded by core/adapters.py)
2. Data Loaders (HIGH PRIORITY)
Current State:
- `core/data_loader.py` (576 lines) - NEW Week 3 implementation ✅
- `engine/loader.py` (564 lines) - Old implementation
Action:
- ✅ KEEP: `core/data_loader.py`
- ❌ DEPRECATE: `engine/loader.py` → Move to `_deprecated/`
3. Configuration (MEDIUM PRIORITY)
Current State:
- `config.py` (510 lines) - NEW Week 2 unified config ✅
- `response/config.py` (deprecated marker) ✅
Action:
- ✅ KEEP: `config.py`
- ✅ ALREADY DEPRECATED: `response/config.py`
4. Coordinates (GOOD STATE)
Current State:
- `core/coordinates.py` (827 lines) - NEW Week 2 unified ✅
- `transformation/coordinate.py` - Transform operations
-No conflicts
Action:
- ✅ KEEP BOTH: Different purposes
5. Models (CRITICAL - Pydantic v2)
Current State:
- `models/*.py` - All use Pydantic v1 `@root_validator`
- Blocking all imports from `dlm` package
Action:
- 🚨 FIX IMMEDIATELY: Update to Pydantic v2
- Change `@root_validator` → `@model_validator`
- Add `skip_on_failure=True` or migrate properly
---
File Size Refactoring
Files Exceeding 1000 Lines (Should be <500)
| File | Lines | Action |
|---|---|---|
| `inference/artificial.py` | 3691 | Split into 6-7 modules |
| `inference/prompt.py` | 2152 | Split into 4-5 modules |
| `response/links.py` | 2083 | Split by link type |
| `response/vangaurd/motion.py` | 1782 | Split by motion type |
| `relationship/graph.py` | 1639 | Split graph operations |
| `engine/embedder.py` | 1604 | DEPRECATE (use core/) |
| `response/system.py` | 1520 | Split by system type |
| `utils/legacy_utils.py` | 1518 | DEPRECATE/MIGRATE |
| `inference/generator.py` | 1432 | Split generators |
| `services/metrics.py` | 1389 | Split metric types |
| `base.py` | 1218 | Split base classes |
| `relationship/merger.py` | 1146 | Refactor mergers |
| `relationship/estimator.py` | 1064 | Refactor estimators |
| `engine/builder.py` | 1004 | Review necessity |
| `relationship/base.py` | 999 | Already complex |
---
Proposed New Structure
dlm/
├── core/ # Core functionality (Week 2-3)
│ ├── __init__.py
│ ├── coordinates.py ✅ Keep
│ ├── embeddings.py ✅ Keep
│ ├── data_loader.py ✅ Keep
│ ├── adapters.py ✅ Keep
│ └── ircp/ ✅ Keep
│
├── training/ # Training pipeline (Week 3)
│ ├── __init__.py
│ ├── pipeline.py Merge from pipeline/
│ ├── checkpoints.py From pipeline/checkpoint_manager.py
│ ├── data.py From pipeline/data_pipeline.py
│ └── config.py Training-specific config
│
├── evaluation/ # Evaluation & metrics
│ ├── __init__.py
│ ├── metrics.py ✅ Keep
│ ├── validators.py ✅ Keep
│ └── quality.py NEW: From services/metrics.py
│
├── explainability/ # Coordinate explainability
│ ├── __init__.py
│ ├── analyzer.py ✅ Keep
│ ├── debugger.py ✅ Keep
│ └── visualizer.py ✅ Keep
│
├── inference/ # LLM inference (REFACTOR)
│ ├── __init__.py
│ ├── manager.py ✅ Keep (ChainManager)
│ ├── generator/ NEW SUBDIR
│ │ ├── __init__.py
│ │ ├── base.py From generator.py (split)
│ │ ├── artificial.py From artificial.py (split)
│ │ └── cloud.py From cloud_manager.py
│ ├── prompts/ NEW SUBDIR
│ │ ├── __init__.py
│ │ ├── templates.py From prompt.py (split)
│ │ ├── manager.py From prompt_manager.py
│ │ └── builder.py From prompt.py (split)
│ ├── session.py ✅ Keep
│ ├── state.py ✅ Keep
│ └── validator.py ✅ Keep
│
├── relationships/ # Relationship estimation
│ ├── __init__.py
│ ├── graph/ NEW SUBDIR
│ │ ├── __init__.py
│ │ ├── builder.py From graph.py (split)
│ │ ├── traversal.py From graph.py (split)
│ │ └── operations.py From graph.py (split)
│ ├── estimator.py Keep but refactor
│ ├── merger.py Keep but refactor
│ └── base.py ✅ Keep
│
├── response/ # Response generation techniques
│ ├── __init__.py
│ ├── builder.py ✅ Keep
│ ├── factory.py ✅ Keep
│ ├── system.py Refactor/split
│ ├── links/ NEW SUBDIR
│ │ ├── __init__.py
│ │ ├── builder.py From links.py (split)
│ │ ├── types.py From links.py (split)
│ │ └── validator.py From links.py (split)
│ ├── embedding_provider.py ✅ Keep
│ ├── validators.py ✅ Keep
│ └── techniques/ Rename from vangaurd/
│ ├── __init__.py
│ ├── synthesis/ From vangaurd/synth/
│ ├── personas/ From vangaurd/word_weaver/
│ └── ... (organize)
│
├── transformation/ # Coordinate transformations
│ ├── __init__.py
│ ├── coordinates.py ✅ Keep
│ ├── tree.py ✅ Keep
│ └── ... (keep)
│
├── models/ # Data models (FIX PYDANTIC)
│ ├── __init__.py
│ ├── message.py Fix Pydantic v2
│ ├── chain.py Fix Pydantic v2
│ ├── generation.py Fix Pydantic v2
│ └── ... (fix all)
│
├── callbacks/ # Event callbacks
│ ├── __init__.py
│ ├── base.py ✅ Keep
│ ├── manager.py ✅ Keep
│ └── streaming.py ✅ Keep
│
├── services/ # Utility services
│ ├── __init__.py
│ ├── preprocessing.py ✅ Keep
│ ├── temporal.py ✅ Keep
│ ├── context.py ✅ Keep
│ └── split.py ✅ Keep
│
├── utils/ # Utilities
│ ├── __init__.py
│ └── logger.py ✅ Keep (new)
│
├── _deprecated/ # OLD CODE (don't break imports yet)
│ ├── engine/
│ │ ├── embedder.py From engine/
│ │ ├── loader.py From engine/
│ │ └── ircp_embedder.py From engine/
│ ├── legacy_utils.py From utils/
│ └── README.md Deprecation notice
│
├── tests/ # Consolidated tests
│ ├── __init__.py
│ ├── core/ Core tests
│ ├── training/ Training tests
│ ├── evaluation/ Evaluation tests
│ ├── explainability/ Explainability tests
│ └── integration/ Integration tests
│
├── examples/ # Usage examples
│ ├── training/
│ ├── explainability/
│ └── inference/
│
├── __init__.py Main package init
├── config.py ✅ Keep (Week 2 unified)
├── base.py Refactor/split
└── interface.py Review
DELETE:
- engine/ (mostly deprecated)
- utils/legacy_utils.py---
Action Plan
Phase 1: Critical Fixes (Immediate)
1. Fix Pydantic v2 Compatibility 🚨
- Update all `@root_validator` → `@model_validator`
- Test imports work
- Priority: CRITICAL
2. Mark Deprecated Code ❌
- Move `engine/embedder.py` → `_deprecated/`
- Move `engine/loader.py` → `_deprecated/`
- Move `engine/ircp_embedder.py` → `_deprecated/`
- Move `utils/legacy_utils.py` → `_deprecated/`
- Add deprecation warnings
Phase 2: Consolidate Tests
3. Unify Test Location
- Move `core/tests/` → `tests/core/`
- Ensure all tests pass
- Update imports
Phase 3: Split Large Files
4. Split inference/artificial.py (3691 lines)
- Create `inference/generator/` subdirectory
- Split by generator type
5. Split inference/prompt.py (2152 lines)
- Create `inference/prompts/` subdirectory
- Split templates/manager/builder
6. Split response/links.py (2083 lines)
- Create `response/links/` subdirectory
7. Split other 1000+ line files
- Prioritize by usage frequency
Phase 4: Organize Structure
8. Reorganize response/vangaurd/
- Rename to `response/techniques/`
- Better organization
9. Clean up engine/
- Keep only actively used files
- Move rest to `_deprecated/`
Phase 5: Documentation
10. Update Documentation
- README for new structure
- Migration guide
- API documentation
---
Impact Assessment
### Files to Keep (Week 2-3 NEW)
- ✅ `core/*` (clean, tested)
- ✅ `evaluation/*` (clean, tested)
- ✅ `explainability/*` (clean, tested)
- ✅ `pipeline/*` (clean, tested)
- ✅ `config.py` (unified)
### Files to Deprecate
- ❌ `engine/embedder.py`
- ❌ `engine/loader.py`
- ❌ `engine/ircp_embedder.py`
- ❌ `utils/legacy_utils.py`
### Files to Refactor (Split)
- ⚠️ `inference/artificial.py` (3691 → 500 each)
- ⚠️ `inference/prompt.py` (2152 → 500 each)
- ⚠️ `response/links.py` (2083 → 500 each)
- ⚠️ `response/vangaurd/motion.py` (1782 → 500 each)
### Files to Fix
- 🚨 `models/*.py` (Pydantic v2)
---
Success Metrics
- [ ] All imports work without Pydantic errors
- [ ] No files exceed 1000 lines
- [ ] Clear separation of concerns
- [ ] All tests pass
- [ ] Documentation updated
- [ ] <10
- [ ] Clear deprecation path
---
Timeline
- Phase 1: 2-3 hours (Critical fixes)
- Phase 2: 1-2 hours (Test consolidation)
- Phase 3: 6-8 hours (Split large files)
- Phase 4: 3-4 hours (Reorganize)
- Phase 5: 2-3 hours (Documentation)
Total Estimated: 14-20 hours
---
Status: AUDIT COMPLETE - READY FOR EXECUTION
Promotion Decision
Attach run IDs, datasets, metrics, and reproduction commands.
Source Anchor
Comp-Core/backend/cc-trajectory/legacy/cc-tpo-original/cc-tpo/docs/refactoring/DLM_REFACTORING_AUDIT.md
Detected Structure
Method · Evaluation · Code Anchors · Architecture