Compass Implementation Roadmap

Compass Implementation Roadmap

> From zero to single pane of glass in 12 weeks.

---

Phase A: Local Dashboard (Week 1-2)

Goal: A Next.js dashboard that reads from existing system state and displays everything in one place.

### Week 1: Foundation
- [ ] Initialize Next.js 15 project with App Router, Tailwind, shadcn/ui
- [ ] Create `collect-state.sh` script that gathers:
- `clawdbot cron list` output (pipeline state)
- Service health checks (curl Graph Kernel, RAG++, Clawdbot Gateway)
- `git status` for all Desktop projects
- Skill file listing from `[home-path]`
- Registry.yaml parse
- [ ] Build data layer (`lib/data-sources.ts`) that reads collected state
- [ ] Pipeline Dashboard — Table view of all 23 cron jobs with status, last run, next run
- [ ] Service Status — Health cards for Graph Kernel, RAG++, Orbit, Clawdbot Gateway

### Week 2: Visualization
- [ ] Architecture Graph — D3.js + Dagre visualization of project categories and connections
- [ ] Skill Registry — Browse all 136 skills by category with search
- [ ] Project Health — Git status, last commit, CI status for Desktop projects
- [ ] Auto-refresh via polling (30-second interval)
- [ ] Dark mode (default) with system preference detection
- [ ] Deploy locally with `npm run dev` — accessible on Tailscale

Deliverable: Working local dashboard showing all system state

---

Phase B: Dagster Integration (Week 3-5)

Goal: Proper pipeline orchestration with dependencies, retries, and monitoring.

### Week 3: Dagster Setup
- [ ] Install Dagster locally (`pip install dagster dagster-webserver`)
- [ ] Define Dagster project structure in `compass/dagster/`
- [ ] Create Python wrappers for 5 low-priority pipelines:
- Garden Tender
- HTDS Rebuild
- Daily MFP Post
- Research Digest
- Daydream Scanner
- [ ] Configure retry policies per pipeline
- [ ] Run in shadow mode alongside existing crons
- [ ] Verify Dagit UI is accessible

### Week 4: Dependency Chains
- [ ] Model pipeline dependencies as Dagster assets:
- Chronicle Capture → Chronicle Synthesis → Content Distributor
- Memory Compaction + Project Scanner → Morning Brief
- Memory Compaction → Weekly Memory Review
- CompCore Health → Living Dashboard
- [ ] Migrate medium-priority pipelines (P2):
- Chronicle Synthesis
- Content Distributor
- Kimi Knowledge Pipeline
- Knowledge Sync
- Project Scanner
- Channel Summarizer
- Weekly Memory Review
- [ ] Add sensors for event-driven triggers (file changes, webhook events)

### Week 5: Full Migration
- [ ] Migrate high-priority pipelines (P1):
- Chronicle Capture
- Koji CRM Pulse
- Living Dashboard
- Heartbeat
- Active Task Monitor
- Memory Compaction
- Morning Brief
- [ ] Migrate critical pipelines (P0) with extra monitoring:
- Agent Auto-Continue
- Trajectory Walk v2
- CompCore Service Health
- [ ] Disable corresponding Clawdbot crons
- [ ] Connect Compass dashboard to Dagster API for live pipeline data
- [ ] Configure alerting: pipeline failures → Discord webhooks

Deliverable: All 23 pipelines running in Dagster with dependency management

---

Phase C: Infrastructure & Containerization (Week 6-8)

Goal: Containerize all services, deploy with Docker Compose, prepare Kubernetes path.

### Week 6: Dockerization
- [ ] Write Dockerfiles for:
- Compass Dashboard (Next.js)
- Compass API (tRPC)
- Dagster (webserver + daemon)
- Graph Kernel
- RAG++
- [ ] Create `docker-compose.yaml` for the full stack
- [ ] Configure networking (service discovery via Docker DNS)
- [ ] Volume mounts for persistent state (SQLite, logs)

### Week 7: Terraform IaC
- [ ] Terraform modules for:
- Docker Compose deployment
- Tailscale integration
- DNS/SSL configuration
- Secrets management
- [ ] State management (Terraform Cloud or local backend)
- [ ] Automated `terraform plan` on PR via GitHub Actions

### Week 8: Kubernetes Preparation
- [ ] Evaluate Kubernetes options:
- k3s (lightweight, single-node)
- minikube (development)
- GKE Autopilot (managed, production)
- [ ] Convert Docker Compose → Kubernetes manifests (Helm charts or kustomize)
- [ ] Deploy to k3s locally as proof of concept
- [ ] Document scaling strategy

Deliverable: Full stack running in Docker Compose with Terraform IaC

---

Phase D: Full Platform (Week 9-12)

Goal: Complete Compass with all features, production hardening, and polish.

### Week 9: Advanced Observability
- [ ] Prometheus metrics export from all services
- [ ] Historical metrics storage and trend visualization
- [ ] Resource tracking dashboard (API costs, compute usage, storage)
- [ ] Token usage tracking per agent session
- [ ] Monthly cost breakdown with trend analysis

### Week 10: Protocol Viewer & Audit Trail
- [ ] Protocol parser (reads from emergent-protocols/, PROTOCOLS.md, skill definitions)
- [ ] Interactive protocol flowcharts
- [ ] Audit trail system:
- Log every pipeline run, agent action, manual trigger
- Searchable audit log UI
- Retention policies (90 days default)
- [ ] RBAC foundation (admin/operator/viewer roles)

### Week 11: Polish & Integration
- [ ] Architecture graph enhancements:
- Live data flow animation
- Zoom levels (Galaxy → Constellation → Star)
- Search with instant navigation
- [ ] Pipeline DAG visualization (interactive, not just table)
- [ ] Skill usage heatmap over time
- [ ] Mobile-responsive layout
- [ ] Keyboard navigation and accessibility (WCAG 2.1 AA)

### Week 12: Production Hardening
- [ ] Load testing (dashboard with all data sources active)
- [ ] Error boundary UI (graceful degradation when services are down)
- [ ] Health check endpoint for Compass itself
- [ ] Documentation:
- User guide
- Developer guide (adding new pipelines/services)
- Operations runbook
- [ ] Final security review
- [ ] Launch announcement

Deliverable: Production-ready Compass platform

---

Quick Wins (Can Start Immediately)

These can be done in parallel with Phase A:

1. Mermaid Architecture Diagram — Already done (architecture.mmd)
2. State Collection Script — `collect-state.sh` gathering all system state to JSON
3. Pipeline Dependency Map — Document all 23 pipeline dependencies
4. Skill Category Index — Parse all 136 skill SKILL.md files into a searchable index
5. Service Health Script — Single script that checks all 4 core services

---

Risk Register

---

Success Criteria

### Phase A Complete When:
- Dashboard loads in < 2 seconds
- All 23 pipelines visible with status
- All 4 services show health status
- Architecture graph renders with categories

### Phase B Complete When:
- All 23 pipelines running in Dagster
- Pipeline dependencies enforced (Morning Brief waits for Memory Compaction)
- Retry policies active on all P0/P1 pipelines
- Discord alerts firing on pipeline failures

### Phase C Complete When:
- `docker compose up` starts the full stack
- Terraform manages the deployment
- All services accessible via Tailscale

### Phase D Complete When:
- Observability dashboard shows 30-day history
- Audit trail captures all agent actions
- Protocol viewer renders all documented protocols
- Lighthouse score: Performance ≥ 90, Accessibility ≥ 95