Stage 2: Compound Synthesis -- Beyond Architecture

Stage 2: Compound Synthesis -- Beyond Architecture

Step 1: Ground Truth -- What Beyond Actually Is

Starting from scratch. Beyond is a NUMU FARE package (`numu-beyond`) that:

1. Accepts any sequence of typed events on the bus
2. Groups them into "processes" (identified by a processId)
3. Embeds each event into a vector
4. Computes anticipation scalars over the trajectory
5. Publishes scalar snapshots and intervention signals
6. Enables paradigm adapters (OmniFlow, Draft-and-Prune, Prompt Optimization) to plug in as process types that define how to embed events, interpret scalars, and execute interventions

Beyond is NOT a separate process, NOT a Python sidecar, NOT a Gemini API wrapper. It is a TypeScript package inside the NUMU daemon that operates on bus events in real time.

The core insight from the research: all three paradigms follow the same geometric trajectory (explore -> converge -> lock). Anticipation scalars make this trajectory visible and actionable without domain-specific convergence criteria.

Step 2: TypeScript Scalar Engine (from Path A)

Building on Step 1. The scalar engine is a TypeScript port of `AnticipationGeometry.compute()` from `python/anticipation_geometry/generalized_anticipation.py`.

The port covers 4 scalars (not 7 -- phase_stiffness, novelty, and stability require motion-domain features that don't apply to event trajectories):

commitment(t) = 1 - ||s_t - s_{t-1}|| / max_delta
uncertainty(t) = H(angles to KNN of s_t) / H_max
transition_pressure(t) = dc/dt - du/dt (smoothed)
recovery_margin(t) = 1 - min_dist_to_branching_point / max_range

Key design decisions:
- Trajectory buffer: Ring buffer of the last W=100 embedded event vectors. Older events are discarded (not the full trajectory like the Python version). This bounds memory and computation.
- K-nearest neighbors: K=5 for uncertainty, computed via brute-force (W=100 means only 100*5 distance computations per update -- fast enough in TS).
- Smoothing window: 3-step moving average for transition pressure derivatives.
- Regime classifier: Same thresholds as Python (transition_pressure > 0.5 = transitioning, etc.), configurable via numu.toml.

Why not use the Rust crate via FFI? Because the Rust crate is built for motion windows (SkeletonFrame, MotionWindow, 27-bone FK). The Python generalized version is the right abstraction level, and it's 480 lines of NumPy that map cleanly to TypeScript arrays.

Step 3: Event Embedding Strategy (from Path A + Path E)

Building on Steps 1-2. Events need to become vectors for the scalar engine.

Three embedding tiers (from Path A's hash approach + Path E's Gemini insight):

Tier 1: Structural hash (default, 0ms)
Hash event type + key scalar fields into a 32D vector. Deterministic, instant, zero API calls. Sufficient for detecting gross trajectory changes (event type shifts, timing changes, error spikes).

function structuralEmbed(event: BusEvent): number[] {
  const vec = new Float64Array(32);
  // Dimension 0-7: event type hash (one-hot-ish via murmurhash)
  // Dimension 8-15: numeric field values (normalized)
  // Dimension 16-23: temporal features (inter-event gap, burst density)
  // Dimension 24-31: contextual (correlationId stability, source diversity)
  return Array.from(vec);
}

Tier 2: Gemini Flash embedding (on-demand, ~200ms)
For processes tagged as high-stakes, embed event content via Gemini API. Returns 256D vectors with semantic meaning. Cached per event (events are immutable). Used when structural hash produces ambiguous scalars.

Tier 3: Pre-computed embeddings (when available)
If the event already carries an embedding (e.g., from numu-memory's sentence-transformer pipeline), use it directly. This is free and high-quality but only available for events that pass through the memory system.

The embedding tier is configured per process type in numu.toml:

[beyond.embedding]
default_tier = 1       # structural hash
omniflow_tier = 2      # Gemini for simulation analysis
draftprune_tier = 1    # structural is fine for verify/reject cycles
promptopt_tier = 2     # Gemini for prompt quality assessment

Step 4: Geometric State Machine (from Path F)

Building on Steps 1-3. Each Beyond process has a state machine with 5 states derived from the regime classifier:

EXPLORING  -->  COMMITTING  -->  LOCKED
    ^               |               |
    |               v               v
    +---  TRANSITIONING  <---  (convergence reached)
    |
    +---  STALLED (intervention trigger)

State transitions are governed by scalar thresholds, not domain logic:

State transitions publish `beyond.transition` events. LOCKED publishes `beyond.convergence`. STALLED publishes `beyond.intervention`.

The state machine is the orchestration primitive. Paradigm adapters don't implement their own convergence logic -- they define what convergence MEANS in their domain (OmniFlow: ensemble agreement, Draft-and-Prune: verification pass, Prompt Optimization: quality plateau) and let the state machine detect it geometrically.

Step 5: Paradigm Adapters (from Path C interfaces + Path F process model)

Building on Steps 1-4. Each paradigm adapter implements a `BeyondParadigm` interface:

interface BeyondParadigm {
  type: "omniflow" | "draftprune" | "promptopt";

  // Map a bus event to a trajectory point (embedding)
  embedEvent(event: BusEvent): number[];

  // What to do when the state machine signals intervention
  onIntervene(process: BeyondProcess, scalars: ScalarSnapshot): InterventionAction;

  // What to do when convergence is reached
  onConverge(process: BeyondProcess, scalars: ScalarSnapshot): ConvergenceResult;

  // What to do when the process stalls
  onStall(process: BeyondProcess, scalars: ScalarSnapshot): StallRecovery;

  // Initial bus subscriptions for this paradigm
  subscribedEvents(): string[];
}

OmniFlow adapter:
- `subscribedEvents()`: `["task.dispatch", "task.completed", "task.failed"]`
- `embedEvent()`: Embed task output text via structural hash (Tier 1) or Gemini (Tier 2). Key dimensions: output variance across parallel tasks, error rate, completion time spread.
- `onIntervene()`: When transition_pressure drops (convergence stalling), spawn counterfactual probes -- re-dispatch the same task with modified parameters (different model, different temperature, different prompt prefix).
- `onConverge()`: When all parallel results cluster tightly (low uncertainty), select the centroid result as the answer.
- `onStall()`: If no progress for N events, expand the ensemble (add more parallel tasks) or escalate to human.

Draft-and-Prune adapter:
- `subscribedEvents()`: `["task.completed", "task.failed", "beyond.verify"]`
- `embedEvent()`: Embed verification result as a binary-augmented vector (pass/fail + error type hash + attempt number).
- `onIntervene()`: When recovery_margin drops (getting deeper into a failing branch), inject accumulated feedback into the next generation prompt.
- `onConverge()`: When verification passes (commitment high, uncertainty low), accept the result.
- `onStall()`: If rejection-regeneration loop exceeds 5 cycles without convergence, switch verification strategy or escalate.

Prompt Optimization adapter:
- `subscribedEvents()`: `["beyond.prompt.scored", "beyond.prompt.mutated"]`
- `embedEvent()`: Embed prompt quality scores + structural features (length, section count, example count).
- `onIntervene()`: When transition_pressure spikes (rapid quality changes), increase exploration (more mutations, wider search).
- `onConverge()`: When quality plateaus at a high level (commitment > 0.8, quality > threshold), freeze the best prompt.
- `onStall()`: If quality is low and not improving, restart from a different seed prompt.

Step 6: Bus Integration and Schema (from Path A + Stage 0 constraints)

Building on Steps 1-5. Four new bus event types, registered in `numu-bus/src/schema.ts`:

beyond.trajectory -- Published every N events or T seconds for each active process:

{
  type: "beyond.trajectory",
  processId: string,
  processType: "omniflow" | "draftprune" | "promptopt" | "custom",
  state: "exploring" | "committing" | "locked" | "transitioning" | "stalled",
  scalars: {
    commitment: number,      // [0, 1]
    uncertainty: number,     // [0, 1]
    transitionPressure: number,  // unbounded, typically [-2, 2]
    recoveryMargin: number,  // [0, 1]
  },
  trajectoryLength: number,
  eventCount: number,
  embeddingTier: 1 | 2 | 3,
}

beyond.transition -- Published on state machine transitions:

{
  type: "beyond.transition",
  processId: string,
  fromState: string,
  toState: string,
  trigger: string,  // human-readable reason
  scalars: { ... },
}

beyond.intervention -- Published when a paradigm adapter acts:

{
  type: "beyond.intervention",
  processId: string,
  interventionType: "counterfactual" | "feedback_inject" | "expand" | "restart" | "escalate",
  reason: string,
  action: object,  // paradigm-specific action details
}

beyond.convergence -- Published when a process reaches LOCKED state:

{
  type: "beyond.convergence",
  processId: string,
  processType: string,
  result: object,    // paradigm-specific result
  finalScalars: { ... },
  totalEvents: number,
  durationMs: number,
}

Publishing strategy: `beyond.trajectory` uses `publishLocal()` by default (high frequency, internal only). Transitions, interventions, and convergences use `publish()` (rare, externally relevant). Configurable via `numu.toml [beyond] broadcast_trajectory = false`.

Step 7: Auto-Research -- Self-Monitoring Meta-Trajectory (from Path F)

Building on Steps 1-6. Beyond monitors its own aggregate trajectory.

The meta-trajectory is computed from all active Beyond processes:
- Each active process contributes its latest scalar snapshot
- The meta-scalars are the weighted mean of child process scalars, weighted by process age (newer processes contribute more)
- The meta-trajectory feeds into its own state machine with the same 5 states

When the meta-trajectory enters STALLED:
1. Identify the most-stalled subprocess (lowest commitment + lowest transition_pressure)
2. Publish `beyond.intervention` with type "meta-diverge"
3. The stalled subprocess's paradigm adapter receives `onStall()` and takes corrective action

When the meta-trajectory enters LOCKED:
1. All subprocesses have converged or are converging
2. Publish `beyond.convergence` with aggregate results
3. Clean up process state

This is NOT recursive -- the meta-trajectory is a single level of observation over child processes. It does not observe itself. The meta-state machine uses the same scalar engine as child processes but is not itself a Beyond process.

Step 8: Integration with Existing Infrastructure

Building on Steps 1-7. How Beyond connects to the mesh:

KARL integration: Every `beyond.convergence` and `beyond.intervention` event is logged as a KARL trajectory entry. The scalar history becomes training signal: "when this paradigm entered STALLED at event 23 and was corrected by expanding the ensemble, it converged 15 events later." KARL learns which interventions work for which scalar patterns.

Graph Kernel integration: For processes that need KG-path validation (Draft-and-Prune with knowledge-grounded verification), the adapter calls GK via HTTP (same pattern as `kg_reward.py`). This is a paradigm-specific integration, not a core Beyond feature.

Mesh compute integration: OmniFlow's parallel ensemble can dispatch tasks to different machines via `task.dispatch` with `machineTarget`. The existing fleet topology in numu.toml provides the routing. Beyond doesn't manage machines -- it manages trajectories. Machine allocation is the dispatch layer's job.

Nexus Portal: A new dashboard page `/beyond` shows active processes, their state machine positions, scalar time series, and intervention history. Data comes from `beyond.trajectory` events collected by the existing Nexus event collector.

numu.toml configuration:

[beyond]
enabled = true
window_size = 100        # trajectory buffer length
publish_interval = 10000 # ms between trajectory broadcasts
broadcast_trajectory = false  # use publishLocal for trajectory events
meta_trajectory = true   # enable self-monitoring

[beyond.thresholds]
exploring_to_committing_commitment = 0.5
committing_to_locked_commitment = 0.7
committing_to_locked_uncertainty = 0.3
stall_events = 10        # events with no movement before STALLED
intervention_pressure_threshold = -0.3

[beyond.embedding]
default_tier = 1
gemini_model = "gemini-2.0-flash"
gemini_cache_ttl_ms = 300000  # 5 minutes