Grand Diomande Research · Full HTML Reader

Latent State Representation

Your life trajectory cannot be fully captured by observable variables alone. There's a **hidden structure**—a latent state—that determines how you respond to opportunities, challenges, and actions.

Agents That Account for Themselves research note experiment writeup candidate score 26 .md

Full Public Reader

Latent State Representation

Why Latent States?

Your life trajectory cannot be fully captured by observable variables alone. There's a hidden structure—a latent state—that determines how you respond to opportunities, challenges, and actions.

Think of it like physics:
- Observable: Position, velocity (where you are, how fast you're moving)
- Latent: Energy, momentum, potential (what determines future motion)

In TrajectoryOS:
- Observable: Projects, skills, constraints (what we can measure directly)
- Latent: Your deep capability, direction, internal state (what predicts future evolution)

The Latent Vector (z_t)

We represent your life state at time `t` as a high-dimensional vector:

z_t ∈ ℝ^d  (typically d=64 or d=128)

This vector encodes:
- Skill landscape: What you truly know (beyond self-reported confidence)
- Directional intent: Where you're naturally pulled
- Energy/capacity: Current creative/cognitive reserves
- Constraint burden: Invisible load from obligations
- Growth trajectory: Momentum and rate of change

Why Not Just Use Observables?

Problem: Observable variables are noisy, incomplete, and lag reality.

Example:

Observable: "I learned React 3 months ago"
Reality: But you haven't used it since, so true competency has decayed

Solution: The latent vector `z_t` models the true underlying state, which generates observables through a learned function.

State Space Model

State Evolution:
z_t+1 = f(z_t, u_t, w_t)

Observations:
y_t = g(z_t, ε_t)

Where:
  z_t = latent state at time t
  u_t = user actions (learning, project changes, etc.)
  w_t = external shocks (job offer, injury, etc.)
  y_t = observations (skill evidence, project status, etc.)
  ε_t = observation noise

Transition Function f(·)

How the latent state evolves:

python

z_t+1 = z_t + Δt × dynamics(z_t, u_t) + noise(w_t)

dynamics(z, u) = [
  skill_growth(z, u.practice_time),
  alignment_shift(z, u.project_changes),
  energy_recovery(z, u.rest_time),
  constraint_accumulation(z, u.new_commitments)
]

Key Properties:
- Continuity: Small actions cause small changes (no sudden jumps unless shocked)
- Memory: z_t depends on history (path integral of past actions)
- Attractors: Certain states are stable (flow, burnout, etc.)

Observation Function g(·)

How latent state manifests in observables:

python

# Skill evidence
skill_level_observed = skill_level_true(z) + noise

# Project completion
project_speed = capability(z) × alignment(z) / mass(z)

# Interview responses
verbal_confidence = true_confidence(z) × calibration(user)

Inverse Problem: Given observations y_1:t, infer latent state z_t
- This is a Bayesian filtering problem
- We use particle filters or variational inference

Dimensionality

What Each Dimension Encodes

We learn a structured latent space where dimensions have interpretable meaning:

Example 64D Latent Space:

Dimensions 0-31: Skill embeddings
  - Each skill maps to a region in this subspace
  - Similar skills cluster together
  - Transfer learning = distance in embedding space

Dimensions 32-47: Directional intent
  - Where you're pulled (career, creative, personal)
  - Alignment = coherence of this vector

Dimensions 48-55: Energy/capacity
  - Creative energy, cognitive capacity, physical vitality
  - Thrust depends on this

Dimensions 56-63: Constraint & inertia
  - Hidden load, system complexity, obligations
  - Gravity and Mass depend on this

Visualization (2D Projection)

           High Energy
                 │
                 │
     Creative ────┼──── Technical
                 │
                 │
            Low Energy

By projecting `z` into 2D (using PCA or t-SNE), we can visualize:
- Clusters: Groups of similar life states
- Trajectories: How users move through state space
- Attractors: Common end states (burnout, flow, escape)

Physics Variables as Derived Quantities

The life physics variables (Thrust, Alignment, Gravity, Mass) are not stored directly—they're computed from the latent state:

python

def compute_physics(z: LatentVector) -> LifePhysics:
    """Derive observable physics from latent state"""

    # Extract skill vector
    skill_embedding = z[0:32]
    skill_levels = decode_skills(skill_embedding)

    # Extract alignment
    direction_vector = z[32:48]
    alignment = compute_concentration(direction_vector)  # 0-1

    # Compute thrust
    utilization = estimate_utilization(z)  # from recent activity
    thrust = alignment * sum(skill_levels * utilization)

    # Extract constraints
    constraint_burden = z[56:60]
    gravity = decode_gravity(constraint_burden)

    # Extract mass
    complexity_vector = z[60:64]
    mass = decode_mass(complexity_vector)

    # Escape index
    escape_index = thrust / (gravity * mass)

    return LifePhysics(
        thrust=thrust,
        alignment=alignment,
        gravity=gravity,
        mass=mass,
        escape_index=escape_index
    )

Advantage: The latent state is lower-dimensional and smoother than raw observables, making it easier to model dynamics.

Updating the Latent State

Evidence Integration

When new evidence arrives (interview, artifact, Echelon session), we update the belief about `z`:

python

# Prior belief
p(z_t | y_1:t-1)

# New evidence
y_t = skill_evidence_from_interview

# Likelihood
p(y_t | z_t) = how likely is this evidence given state z?

# Posterior (Bayes rule)
p(z_t | y_1:t) ∝ p(y_t | z_t) × p(z_t | y_1:t-1)

In Practice:
1. Particle Filter: Maintain 1000 samples of z, reweight by likelihood
2. Variational Inference: Optimize Gaussian approximation q(z) ≈ p(z | y)
3. Kalman Filter: If dynamics are linear (they're not, but approximate)

Multi-Modal Fusion

When Echelon activates, we fuse verbal and embodied evidence:

python

# Verbal evidence
y_verbal = "I'm aligned on this project"
p(z | y_verbal) → increases alignment dimension

# Embodied evidence (same time)
y_embodied = EchelonSummary(drift=high, flow=low)
p(z | y_embodied) → decreases alignment dimension

# Combined
p(z | y_verbal, y_embodied) ∝ p(y_verbal | z) × p(y_embodied | z) × p(z)

# Result: Embodied evidence has higher weight (more trustworthy)
# Final belief: alignment is lower than verbal claim

Key Insight: Embodied signals act as ground truth to calibrate latent state, overriding verbal claims when they conflict.

Trajectory Prediction

Once we have `z_t`, we can simulate forward:

python

def predict_trajectory(
    z_current: LatentVector,
    hypothetical_actions: List[Action],
    time_horizon: int = 365  # days
) -> List[LifeState]:
    """Simulate future trajectory"""

    trajectory = []
    z = z_current

    for day in range(time_horizon):
        # Apply action
        u_t = hypothetical_actions[day] if day < len(hypothetical_actions) else None

        # State transition
        z = transition_function(z, u_t, noise=sample_noise())

        # Compute observables
        physics = compute_physics(z)

        trajectory.append(LifeState(
            latentVector=z,
            physics=physics,
            timestamp=now + timedelta(days=day)
        ))

    return trajectory

Use Cases:
- Escape Time Estimation: How long until η > 1?
- Scenario Analysis: What if I take this job?
- Leverage Point Identification: Which action increases η most?

Monte Carlo Simulation

Since dynamics are stochastic, we run many simulations:

python

n_simulations = 1000
escape_times = []

for _ in range(n_simulations):
    trajectory = predict_trajectory(z_current, actions)
    escape_time = first_time_eta_gt_1(trajectory)
    escape_times.append(escape_time)

# Statistics
median_escape = np.median(escape_times)
confidence_90 = np.percentile(escape_times, [5, 95])

Output: "You'll likely reach escape velocity in 6-9 months (90

Learning the Latent Space

Initial Approach: Hand-Crafted

For MVP, we manually define the latent dimensions:
- Map skills to embeddings using pre-trained NLP models
- Alignment = hand-coded based on project similarity
- Energy = heuristic from recent activity

This works but is brittle.

Future: Learned Representation

Train a variational autoencoder (VAE) or state-space model:

python

# Encoder: y_1:t → z_t
z_t = encoder(interview_transcripts, skill_evidence, project_data)

# Decoder: z_t → ŷ_t+1
predicted_evidence = decoder(z_t)

# Loss
loss = reconstruction_loss(y_t+1, predicted_evidence) + KL_divergence(z_t, prior)

Training Data:
- User histories (observables over time)
- Outcomes (did they reach escape velocity?)
- Embodied signals (Echelon sessions)

Result: The model learns a latent space where similar trajectories cluster, enabling transfer learning across users.

Personalization

Each user has a personalized latent space:
- Global structure: Shared across all users
- User-specific: Fine-tuned to individual

This allows:
- Cold start: Use population model for new users
- Personalization: Improve as more data arrives
- Transfer: Skills learned elsewhere inform your model

Interpretability

Latent Space Exploration

User Query: "Why did my escape index drop last month?"

System Response:

Your latent state shifted in the following ways:
- Alignment dimension: 0.82 → 0.65 (project added: crypto bot)
- Energy dimension: 0.70 → 0.58 (less sleep, more stress)
- Skill growth: Python +0.1, but scattered across 5 new frameworks

Root cause: Taking on unaligned project while energy was already low.
Recommendation: Drop crypto bot, focus on core projects.

Latent Interventions

User Action: "What if I improve my sleep?"

System Simulation:

Hypothetical: +2 hours sleep/night for 30 days

Predicted latent shift:
- Energy: 0.58 → 0.75
- Cognitive capacity: +15%
- Stress: 0.62 → 0.45

Predicted physics change:
- Thrust: +8% (more energy → better skill utilization)
- Gravity: -12% (stress reduction)
- Escape index: 0.94 → 1.08 ✅

Conclusion: Sleep improvement pushes you over escape velocity.

Embodied Latent Dimensions (Future)

When Echelon integrates, we add embodied latent dimensions:

Dimensions 64-71: Embodied state
  - Flow propensity (how easily you enter flow)
  - Tension baseline (resting stress level)
  - Movement quality (coordination, grace)
  - Rhythm coherence (internal tempo stability)

These dimensions are only accessible via Echelon—no amount of verbal reporting can reveal them.

Example:

User A: High verbal confidence in choreography
  - Embodied flow propensity: Low (struggles to enter flow)
  - Latent skill (true): Medium

User B: Moderate verbal confidence in choreography
  - Embodied flow propensity: High (enters flow easily)
  - Latent skill (true): High

Conclusion: User B is actually more skilled, despite lower verbal confidence.

Mathematical Formulation

Bayesian State Space Model

Latent Dynamics:
p(z_t | z_{t-1}, u_t) = 𝒩(z_t | f(z_{t-1}, u_t), Q)

Observation Model:
p(y_t | z_t) = 𝒩(y_t | g(z_t), R)

Posterior Inference:
p(z_t | y_1:t) = ∫ p(y_t | z_t) p(z_t | z_{t-1}, u_t) p(z_{t-1} | y_1:t-1) dz_{t-1}

Challenges:
- Non-linear f(·) and g(·) → No closed-form
- High-dimensional z → Curse of dimensionality
- Sparse, noisy observations → Uncertainty

Solutions:
- Particle filters for inference
- Neural networks for f(·) and g(·)
- Variational methods for tractability

Physics Constraint

We enforce that derived physics must match known relationships:

Constraint:
η(z) = T(z) / (G(z) × M(z))

This ensures internal consistency.

Conclusion

The latent state representation provides:

1. Unified Model: All evidence (verbal, behavioral, embodied) updates the same z
2. Predictive Power: Simulate future trajectories
3. Interpretability: Understand why η changed
4. Personalization: Each user's z evolves uniquely
5. Embodied Grounding: Echelon signals anchor latent state in physical reality

By modeling life as a latent dynamical system, we go beyond surface-level metrics to capture the true underlying state that determines your trajectory.

---

Next: See [Service Architecture](../architecture/services.md) to understand how this is implemented in the system.

Promotion Decision

Attach run IDs, datasets, metrics, and reproduction commands.

Source Anchor

Comp-Core/backend/cc-trajectory/docs/concepts/latent-state.md

Detected Structure

Method · Evaluation · Architecture