Stage 3: Expand + Master Plan -- AgentOS Cognitive Synchronization

Stage 3: Expand + Master Plan -- AgentOS Cognitive Synchronization

---

3a. Risk Audit

CRITICAL RISKS

#### R1: Interrupt File Race Condition
Failure scenario: PostToolUse hook and SessionEnd hook fire within milliseconds of each other for the same pane. Both try to write `/tmp/pane_interrupts/ttys007.json`. One overwrites the other. The SessionEnd signal (high priority) is lost because PostToolUse (low priority) clobbered it.

Probability: Medium (hooks fire frequently, timing overlap is plausible)
Impact: High (missed SIG_SESSION_END means the orchestrator thinks the pane is still working for up to 5 minutes)

Mitigation: Use append-mode interrupt directory, not single file per pane:

/tmp/pane_interrupts/ttys007/
    1710100000_0x01_tool_active.json
    1710100001_0x03_session_end.json

Each interrupt is a separate file. The orchestrator reads ALL files in the directory, sorts by timestamp, takes the highest-priority one, and deletes consumed files.

Validation: Write a test that fires 100 concurrent interrupts to the same TTY directory and verifies zero data loss.

---

#### R2: Drift Meter Threshold Produces False Positives at High Pane Count
Failure scenario: With 40 panes, each emitting SIG_TOOL_ACTIVE (drift += 0.02), the aggregate drift noise triggers sync pulses continuously. The system enters a "sync storm" where every cycle fires a pulse, defeating the purpose of event-driven sync.

Probability: Medium-High (at 40 panes with 30s cycle, each pane emits ~2 tool events per second = 80 events/30s = 1.6 drift added per cycle across fleet)
Impact: High (sync storm degrades to worse than temporal polling)

Mitigation: Drift meters are PER-PANE, not global. A sync pulse fires only when a SINGLE pane crosses threshold. SIG_TOOL_ACTIVE should NOT increment drift at all -- it is a "heartbeat" signal, not a drift signal. Revise weights:

weights = {
    0x01: 0.00,    # tool_active -- NO drift, just "I'm alive"
    0x02: 0.40,    # context_full
    0x03: 0.60,    # session_end
    0x04: 0.80,    # error_stuck
}

Only high-value interrupts (context_full, session_end, error_stuck) contribute to drift. Tool activity is tracked separately for idle detection but does NOT feed the drift meter.

Validation: Simulate 40 panes with realistic interrupt patterns (1 tool_use every 15s, 1 context_full per hour, 2 session_ends per hour). Verify sync pulse frequency stays below 1 per 5 minutes on average.

---

#### R3: Content Fingerprint File Path Extraction Misses Dynamic Paths
Failure scenario: Claude Code constructs file paths via variables (`${HOME}/projects/...`) or reads them from JSON/YAML. The regex `(?:/[a-zA-Z0-9._-]+)+\.[a-zA-Z]{1,10}` misses these. Two panes working on the same project are not flagged as duplicates.

Probability: High (variable-based paths are common in terminal output)
Impact: Medium (duplicate detection is a nice-to-have, not a safety-critical feature)

Mitigation: Multi-strategy extraction:
1. Regex for literal paths (current)
2. Match project names from pane registry (from pane_registry.json project field)
3. Match known project directories from AGENTS.md or ARCHITECTURE.md
4. Fall back to keyword overlap on non-path tokens

Validation: Run fingerprint extraction on 20 saved terminal output snapshots from /tmp/pane_context/ and manually verify file path coverage.

---

MEDIUM RISKS

#### R4: NUMU Bus Schema Migration Requires Downtime
Failure scenario: Adding `sync.*` message types to numu-bus/src/schema.ts requires TypeScript compilation and daemon restart. During restart (10-30s), all NUMU consumers lose their WebSocket connection. Events emitted during this window are lost.

Probability: Certain (schema change requires recompile)
Impact: Low-Medium (30s of lost events is recoverable; the fallback timer catches it)

Mitigation:
- The sync.* messages are ADDITIONAL types, not modifications to existing ones. Existing consumers ignore unknown message types.
- Restart during low-activity window (US nighttime / Mohamed sleeping)
- NUMU bus clients already reconnect on disconnect (built into the WS client)

Validation: Send a burst of 100 events spanning the restart window. Verify the reconnected client receives events after restart.

---

#### R5: Haiku Skip Rate Causes Missed Context Exhaustion
Failure scenario: The "auto-compact" string detection in PostToolUse output fails because Claude changes its compaction message format. No SIG_CONTEXT_FULL interrupt fires. The orchestrator relies on Haiku to detect it, but Haiku was skipped because recent tool activity was detected. The context-exhausted pane sits unnoticed.

Probability: Low-Medium (Claude's compaction message has been stable but is not contractual)
Impact: Medium (pane wastes time in degraded context until the next Haiku check or the 5-min fallback)

Mitigation:
- Maintain a LIST of compaction indicators, not a single string: "auto-compact", "compressed prior", "context window", "messages compacted"
- Add a "time since last interrupt" counter. If a pane has been emitting SIG_TOOL_ACTIVE for >15 minutes continuously, force a Haiku check regardless (suspiciously long tool activity = possible context degradation)
- KARL trajectories that end with context exhaustion can retroactively identify the undetected pattern

Validation: Grep the last 30 days of terminal logs for compaction messages and verify the indicator list catches all variants.

---

#### R6: Priority Affinity Matching Creates Project Starvation
Failure scenario: OpenClawHub has 10 pending tasks and 5 panes with OpenClawHub affinity. feed-hub has 3 pending tasks but 0 panes with feed-hub affinity. All idle panes get assigned OpenClawHub tasks. feed-hub tasks starve indefinitely.

Probability: Medium (project clustering is a real pattern in the fleet)
Impact: Medium (some projects get delayed, but nothing breaks)

Mitigation: Add a starvation detector. If a task has been pending for longer than MAX_TASK_WAIT (30 minutes), boost its priority by 0.30, overriding affinity. This guarantees every task eventually gets scheduled regardless of pane affinity.

if task.get("created_at"):
    age_min = (time.time() - task["created_at"]) / 60
    if age_min > 30:
        source_weight += 0.30  # starvation boost

Validation: Simulate a workload where one project has 80

---

#### R7: Compass Review Prompt Generates Hallucinated Recommendations
Failure scenario: Haiku misinterprets the fleet snapshot and recommends reprioritizing a task that does not exist, or alerting about a non-existent duplicate. If auto-execution were enabled, this would cause harm.

Probability: Medium (Haiku 4.5 can hallucinate on structured data)
Impact: Low (Compass is advisory-only -- logged but not executed)

Mitigation:
- Compass responses are LOGGED ONLY, never auto-executed
- Include a "valid_task_ids" field in the prompt so Haiku can only reference real tasks
- Parse Compass output and validate referenced TTYs/task_ids against actual fleet state before logging

Validation: Run 50 Compass reviews on synthetic fleet snapshots and measure hallucination rate on task_id and TTY references.

---

LOW RISKS

#### R8: SyncMetrics Counter Overflow
Impact: Negligible. Counters are Python ints (arbitrary precision). Reset on daemon restart.

#### R9: File Path Regex Matches Non-File Strings
Impact: Low. False positive file paths in content fingerprint cause slightly noisier duplicate detection. No behavioral consequence.

#### R10: Interrupt File Accumulation on Disk
Impact: Low. Each interrupt file is <1KB. At 100 files/minute, 1 day = 144K files = ~144MB. Orchestrator consumes (deletes) files on each cycle. Risk only if orchestrator is down for extended period.

Mitigation: A cron job that purges interrupt files older than 1 hour.

---

3b. Expanded Specifications

Spec 1: Interrupt Signal Emitter (post_tool_hook.py modification)

Location: `[home-path]`
New function: `_emit_interrupt(tty, vector, signal_name, metadata)`
Interrupt directory: `/tmp/pane_interrupts/{tty_safe}/`
File naming: `{unix_ts_ms}_{vector_hex}_{signal_name}.json`

Interrupt JSON schema:

{
    "vector": 3,
    "signal": "SIG_SESSION_END",
    "tty": "/dev/ttys007",
    "machine_id": "mac1",
    "project": "OpenClawHub",
    "timestamp": 1710100001.234,
    "metadata": {
        "duration_sec": 1234,
        "tool_count": 47,
        "exit_reason": "natural"
    }
}

Error detection state (per-pane, persisted in `/tmp/pane_error_state/{tty_safe}.json`):

{
    "consecutive_failures": 3,
    "last_failure_ts": 1710100000.0,
    "last_success_ts": 1710099900.0,
    "failure_tools": ["Bash", "Bash", "Bash"]
}

Threshold: 3 consecutive Bash failures within 120s triggers SIG_ERROR_STUCK.

Context exhaustion indicators:

COMPACTION_INDICATORS = [
    "auto-compact",
    "compressed prior messages",
    "context window",
    "messages compacted",
    "conversation too long",
    "token limit",
]

Match is case-insensitive substring search on tool output.

Lines of code: ~60 new lines in post_tool_hook.py, ~30 new lines in session_end_hook.py

---

Spec 2: Drift Meter Engine

Location: New file `[home-path]`
Class: `DriftMeter` (one instance per tracked pane)
Manager class: `DriftFleet` (manages all meters, checks thresholds)

class DriftFleet:
    """Manages per-pane drift meters and fires sync pulses."""

    def __init__(self, decay_lambda: float = 0.005, threshold: float = 0.8):
        self.meters: dict[str, DriftMeter] = {}
        self.threshold = threshold
        self.decay_lambda = decay_lambda

    def ingest_interrupt(self, tty: str, vector: int):
        """Process an interrupt from a pane."""
        if tty not in self.meters:
            self.meters[tty] = DriftMeter(decay_lambda=self.decay_lambda)
        self.meters[tty].on_interrupt(vector)

    def check_pulses(self) -> list[str]:
        """Return list of TTYs that have crossed the drift threshold."""
        pulsing = []
        for tty, meter in self.meters.items():
            if meter.should_pulse(self.threshold):
                pulsing.append(tty)
                meter.reset()  # Clear drift after pulse fires
        return pulsing

    def prune_stale(self, max_age_sec: float = 3600):
        """Remove meters for panes that no longer exist."""
        now = time.time()
        stale = [tty for tty, m in self.meters.items()
                 if now - m.last_update > max_age_sec]
        for tty in stale:
            del self.meters[tty]

Drift weights (revised from R2 mitigation):

DRIFT_WEIGHTS = {
    0x01: 0.00,    # SIG_TOOL_ACTIVE: no drift, just alive signal
    0x02: 0.40,    # SIG_CONTEXT_FULL: needs attention
    0x03: 0.60,    # SIG_SESSION_END: pane is now free
    0x04: 0.80,    # SIG_ERROR_STUCK: critical, needs intervention
}

Decay constant: lambda = 0.005 (half-life ~138 seconds). This means:
- A SIG_SESSION_END (0.60) decays to 0.30 in ~138s
- Without any additional events, threshold (0.8) is only crossed by SIG_ERROR_STUCK alone or SIG_SESSION_END + SIG_CONTEXT_FULL within 2 minutes

Lines of code: ~80 lines (drift_meter.py)

---

Spec 3: Content Fingerprinter

Location: New function in `cortex_orchestrator.py` (not a separate file -- too small)
Function: `_compute_content_fingerprint(terminal_output: str) -> dict`

Multi-strategy extraction:

def _compute_content_fingerprint(terminal_output: str) -> dict:
    """Extract content fingerprint from terminal output."""
    lines = terminal_output.strip().split("\n")

    # Strategy 1: Literal file paths
    file_re = re.compile(r'(?:/[a-zA-Z0-9._-]+){2,}\.[a-zA-Z]{1,10}')
    files = set(file_re.findall(terminal_output))

    # Strategy 2: Project directory names
    proj_re = re.compile(
        r'(?:Desktop|projects|flows|monitoring|\.claude)/([a-zA-Z0-9_-]+)'
    )
    projects = set(proj_re.findall(terminal_output))

    # Strategy 3: Content hash of last 20 lines
    tail = "\n".join(l.strip() for l in lines[-20:] if l.strip())
    content_hash = hashlib.sha256(tail.encode()).hexdigest()[:16]

    return {
        "files": sorted(files)[:10],
        "projects": sorted(projects)[:5],
        "content_hash": content_hash,
    }

Duplicate detection (integrated into sense phase):

def _detect_duplicates(self, fingerprints: dict[str, dict]) -> list[dict]:
    """Detect panes working on same files. O(n) via hash grouping."""
    file_panes: dict[str, list[str]] = {}
    for tty, fp in fingerprints.items():
        for f in fp.get("files", []):
            file_panes.setdefault(f, []).append(tty)

    return [
        {"file": f, "panes": panes}
        for f, panes in file_panes.items()
        if len(panes) > 1
    ]

Lines of code: ~50 lines (added to cortex_orchestrator.py)

---

Spec 4: NUMU Sync Domain Schema

Location: `[home-path]`
New types: SyncDriftReport, SyncPulse, SyncDuplicate (3 types)
New domain number: 7 (after health domain 4, thread domain 3... existing numbering)

Full TypeBox definitions as specified in compound Step 5. Added to the existing Validators map:

export const Validators = {
    // ... existing validators ...
    "sync.drift_report": TypeCompiler.Compile(SyncDriftReport),
    "sync.pulse": TypeCompiler.Compile(SyncPulse),
    "sync.duplicate_detected": TypeCompiler.Compile(SyncDuplicate),
};

Lines of code: ~60 lines in schema.ts

---

Spec 5: cortex_orchestrator.py Integration

Modified phases in the heartbeat cycle:

1. PRE-SENSE (new): Read interrupt directory, feed DriftFleet, check for sync pulses
2. SENSE: Existing pane discovery + NEW content fingerprinting on sync pulse or every 5th cycle
3. CLASSIFY: Existing Haiku classification WITH interrupt-skip logic (Step 2 from compound)
4. MATCH (modified): Priority affinity matching replaces longest-idle-first (Step 7 from compound)
5. INJECT: Existing injection logic (unchanged)
6. CHECK: Existing invariant checks + NEW duplicate detection logging
7. ADAPT: Existing interval adaptation + NEW SyncMetrics recording
8. COMPASS (new, periodic): Every 10 minutes, generate fleet snapshot and send to Haiku for strategic review

New instance variables on CortexOrchestrator class:

self.drift_fleet = DriftFleet()
self.sync_metrics = SyncMetrics()
self.fingerprints: dict[str, dict] = {}
self.recent_duplicates: list[dict] = []
self.compass_last_run: float = 0

Estimated change: ~200 lines modified/added in cortex_orchestrator.py

---

3c. Master Execution Checklist

Wave 0: Foundation (Day 1-2)

Wave 0 Gate: drift_meter.py passes all unit tests. Interrupt directories exist.

---

Wave 1: Interrupt Emitter (Day 2-3)

Wave 1 Gate: All 4 interrupt types fire correctly. Existing hook functionality (logging, KARL, cortex) unbroken.

---

Wave 2: Orchestrator Integration (Day 3-5)

Wave 2 Gate: cortex_orchestrator.py runs stably for 1 hour with interrupt-driven classification. Haiku skip rate >50

---

Wave 3: Priority Matching + NUMU (Day 5-7)

Wave 3 Gate: Priority matching assigns tasks by affinity. sync.* events flow through NUMU. Full interrupt-to-sync pipeline works end-to-end.

---

Wave 4: Compass + Polish (Day 7-9)

Wave 4 Gate: Compass fires every 10 minutes with useful output. Grafana dashboard shows sync metrics. Cleanup cron is running.

---

Wave 5: Observation + Calibration (Day 9-14)

Wave 5 Gate: 5 days of clean operation. Metrics validated. Drift threshold calibrated. Compass quality assessed.

---

Kill Criteria

1. **If Haiku skip rate < 30

2. If sync pulse frequency > 10/hour after Wave 3: Drift threshold is too low or drift weights are too high. Recalibrate before proceeding.

3. If cortex_orchestrator.py cycle time increases by >2x after Wave 2: The interrupt reading and fingerprinting overhead is too high. Profile and optimize or remove the fingerprinting (it is the most expensive new operation).

---

Summary Statistics

---

Architecture Diagram (Final State)

                           ┌───────────────────────┐
                           │    NUMU Bus (:7890)    │
                           │   + sync.* domain      │
                           └───────────┬───────────┘
                                       │
                    ┌──────────────────┼──────────────────┐
                    │                  │                   │
             ┌──────▼──────┐   ┌──────▼──────┐   ┌──────▼──────┐
             │ Mac1 Cortex │   │ Mac2 Cortex │   │ Mac3 Cortex │
             │ Orchestrator│   │ Orchestrator│   │ Orchestrator│
             └──────┬──────┘   └──────┬──────┘   └──────┬──────┘
                    │                  │                   │
         ┌──────────┼──────┐          │                   │
         │          │      │          │                   │
    ┌────▼──┐ ┌────▼──┐ ┌─▼───┐ ┌───▼───┐          ┌───▼───┐
    │Pane A │ │Pane B │ │... N│ │Pane M │          │Pane P │
    │       │ │       │ │     │ │       │          │       │
    │ Hooks │ │ Hooks │ │Hooks│ │ Hooks │          │ Hooks │
    └───┬───┘ └───┬───┘ └──┬──┘ └───┬───┘          └───┬───┘
        │         │        │        │                   │
        ▼         ▼        ▼        ▼                   ▼
    /tmp/pane_interrupts/{tty}/ (per-pane interrupt directory)
        │
        ▼
    ┌───────────────┐     ┌──────────────┐     ┌──────────────┐
    │ DriftFleet    │────>│ Content      │────>│ Priority     │
    │ (per-pane     │     │ Fingerprint  │     │ Affinity     │
    │  drift meters)│     │ (file paths) │     │ Matching     │
    └───────────────┘     └──────────────┘     └──────────────┘
        │                                           │
        │ threshold exceeded                        │ best (task, pane) pair
        ▼                                           ▼
    ┌───────────────┐                          ┌──────────────┐
    │ Sync Pulse    │                          │ Inject        │
    │ (event-driven)│                          │ (clipboard)   │
    └───────┬───────┘                          └──────────────┘
            │
            ▼
    ┌───────────────┐     ┌──────────────┐
    │ SyncMetrics   │────>│ Prometheus   │──> Grafana
    │ (counters)    │     │ :9131        │
    └───────────────┘     └──────────────┘
            │
            │ every 10 min
            ▼
    ┌───────────────┐
    │ Compass Review│
    │ (Haiku 4.5,   │
    │  advisory)    │
    └───────────────┘

---

Key Insight from This Evolution

The AgentOS paper's most valuable contribution is NOT the S-MMU (which requires internal state access we do not have) but the drift-triggered sync model. Replacing temporal polling with event-driven sync, where the events come from the agent's own hooks, bridges the gap between the theoretical AgentOS and our practical multi-process architecture. The interrupt-first classification (Step 2) is the foundation everything else depends on -- it makes the system both faster and cheaper simultaneously, which is rare. The content fingerprinting and duplicate detection (Step 4) are the highest-ROI semantic additions because they solve the concrete "two panes editing the same file" problem without requiring expensive embedding models.