visualize

If you want the audience—and yourself—to feel the shape of Echelon’s inner life as you play, you have to turn an invisible vector (x_t^*\inR^D) into a picture that moves with the same inevitability as the music. The challenge is not just to plot numbers; it’s to build a visual grammar where geometry equals meaning, where distance means “more different,” curvature means “changing intention,” thickness means “tension,” and motion means “you.” The trick is to choose a projection that preserves the two invariants your engine already enforces: contraction in the fast loop and phase alignment in the scheduler. When you respect those invariants visually, the image becomes a second instrument—honest, rhythmic, and interpretable at a glance.

Start by deciding what the axes should mean. You do not want an arbitrary 2D scatter that jitters because the optimization changed its mind. You want a stable map that you can revisit next week and still recognize. There are two good ways to get that stability. One is to learn, offline, a parametric projector (f_\phi:R^D\toR^2) on your rehearsal data—a tiny two-layer network trained to preserve local neighborhoods the way UMAP does—then freeze its weights and run it in real time. The other is to compute a fixed linear basis with PCA on your rehearsals, take the top two or three components, and project live latents onto that plane. The first gives you more curvature and semantic separation; the second gives you maximum predictability with almost no compute. Either way, once you freeze (f_\phi) or the PCA matrix, the map stops drifting; a region that meant “hand-driven shimmer” last month still means it today.

Now tie the map to rhythm. Phase is already a circular variable in Echelon, a number (\psi_t) between 0 and (2\pi) that tells you where inside the beat you are. If you draw your manifold as a flat scatter, phase is homeless. If you draw it on a ring, phase feels like home. A simple and powerful visual is to embed (\psi_t) as angle and latent energy (|x_t^*|) as radius: a polar plot where each frame lands on a circle and the point breathes in and out with effort. That picture alone—the wheel spinning, the comet tail fading behind the head—tells you whether your body is in time. If you want the full manifold and the phase wheel at once, you can lift the 2D projector into a cylinder: use (f_\phi(x)) for the floor plan and draw a thin halo around the current point whose hue rotates with (\psi_t). The halo locks the picture to the beat; the floor plan tells you what kind of motion you’re in.

Contraction belongs in the picture as thickness. The fast loop’s residual (r_t=|x_{t}^{(k+1)}-x_{t}^{(k)}|) is a measure of how hard the solver is working. Draw the live point with an outer glow whose radius or intensity scales with (r_t). When the system is confident, the point is sharp. When evidence conflicts or you deliberately loosen the solver, the aura swells and pulses. That glow is your “strain” meter, and it can be musically mapped as well: the same value that makes the point breathe can drive a tremolo or a filter flutter, so ears and eyes agree about the instrument’s state.

Phrase should be visible as gravity. The slow equilibrium (y_b^) is not just a number; it’s the moving anchor that keeps the short-term reflex from wandering. If you train the projector on pairs ((x_t^, y_b^)) you can learn a second head (g_\psi:R^{D_s}\toR^2) that places the slow state in the same picture. In performance you then draw a translucent attractor disk at (g_\psi(y_b^)) and a spring from the live point to that disk. As the bar evolves, the disk moves; the spring lengthens and shortens with your coupling energy (\phi). When you’re building toward a drop, the attractor drifts outward and the spring tightens; when you release, it snaps inward. You have literally drawn the coupling term: you can see the “intention” that the slow map is exerting on the reflex and how much slack remains.

Localization—where on the body the energy lives—deserves its own layer. Your per-limb energy heads are scalars (\epsilon^{(\ell)}) with immediate physical meaning. Draw them not as numbers but as small satellites orbiting the live point, one per limb, with size or opacity proportional to (\epsilon^{(\ell)}) and angular offset fixed by a body schema (left hand satellites at 210°, right hand at 330°, torso at 90°, etc.). When your arms carry the groove, two bright moons flare beside the head; when your core drives a lift, the top satellite swells. This keeps the main manifold uncluttered while letting the viewer read embodied causality: which part of you is moving the sound right now.

Those are the primitives, but the picture has to move like music. Don’t update at 60 Hz with raw points; that will jitter. Sample the latent at your control rate, say 120 Hz or one update per audio block, and apply a short exponential smoothing in screen space so the path leaves a clean tail. Fade the tail over one bar so phrases appear as arcs you can read in retrospect. When tempo changes, change the fade constant so a bar still looks like a bar. Avoid alpha blending that saturates; write your trails into an offscreen accumulation buffer so you can blur them without starving the GPU. Keep the rendering loop on its own thread; the audio thread must never notice the picture exists.

An atlas completes the mind’s map. Before the show, run your rehearsals through the projector and compute a density map of the manifold—a soft heatmap where you have historically spent time. Overprint the live trace on that atlas, and label a few regions with names you care about: “half-time sway,” “double-time shimmer,” “drop wind-up.” Those labels aren’t magic; they’re simply clusters you’ve annotated from your own sets. When you play, you can see whether you’re revisiting a familiar region or carving new paths. If you want that atlas to breathe, compute a small k-NN neighborhood around the live point in your rehearsal data and flash thumbnails of the phrase segments you played nearest to here last week. The visual becomes a memory prosthesis: the engine isn’t just keeping time; it’s reminding you of how you used to move through this part of the landscape.

There are two extremes worth exploring as alternate views. One is the “phase torus”: draw a donut in 3D where the big circle is bar position and the small circle is beat phase. Map a projection of (x_t^) onto the donut’s surface and let the point run laps. That picture is mesmerizing on a projector and reveals swing and drift instantly; you can see microphase lead or lag as the point rides the inside or outside of the torus. The other is the “vector field”: instead of a point, draw arrows showing the instantaneous velocity (x_t^) in screen space. This is useful in the studio when you want to tune solver parameters: high-frequency jitter means your step size is too large; sluggish arrows mean contraction is too tight for the movement you’re capturing. On stage, you reduce it back to the comet tail.

On the implementation side, the pipeline is straightforward and safe. The control thread computes the projection (p_t=f_\phi(x_t^)) and the derived quantities (\psi_t, |x_t^|, r_t, {\epsilon^{(\ell)}}, g_\psi(y_b^*)) at the same rate you publish `FastState` to the mapper. It writes a compact struct—two floats for position, one for phase, one for energy, one for residual, a handful for limb energies, two for the attractor—into a bounded ring buffer shared with the renderer. The renderer (built on `wgpu` with an immediate-mode UI like egui for overlays) pops samples, interpolates them at monitor refresh, and draws with one or two full-screen fragment shaders: a polar wheel or atlas background blended with the trail and satellites. There are no blocking calls crossing into the audio world; if the renderer falls behind, it drops frames and the sound continues.

The beauty of this approach is that the visuals are not decoration; they are a second reading of the same contract that keeps the music coherent. The contraction glow is the same residual that can modulate tremolo; the spring to the attractor is the same coupling energy that shapes your crossfades; the satellites are the same limb energies that steer filters and sends; the phase wheel is the same (\psi_t) that quantizes events. When the audience sees the halo thicken and hears the texture tremble, or watches the point slide into the “drop” region as the system opens the fader, they are witnessing the mathematics you designed made legible. And when you take the engine into the studio to adjust solver gain or scheduler windows, the same picture becomes your oscilloscope: you can see whether the latent manifold is smooth, whether phases line up across tempo changes, whether slow-state anchors sit where you expect. That is what “visualizing the latent manifold” should mean in Echelon: one coherent moving picture in which rhythm and geometry are the same thing.

What I was describing there is the recipe for building the visualization layer that sits on top of your real-time engine—the part that lets you actually see Echelon’s latent manifold breathing in performance. It’s not something mystical; it’s a practical graphics pipeline that you can build the same way you’d build a lightweight visualizer or performance meter.

You start with a continuous stream of latent vectors (x_t^*) coming out of your solver every few milliseconds. Each of those vectors has, say, 32 or 64 dimensions—far too many to plot directly. The job of the projector (f_\phi) or the PCA basis is to compress those vectors into two coordinates that can be drawn on screen, and to do so in a way that doesn’t change its mind from one session to the next.

Here’s what that looks like in practice:

1. Gather rehearsal data. Record a few minutes of live motion-to-latent data while you perform. Store all the (x_t^*) vectors.

2. Compute a stable projection.

• If you go with the PCA approach, load that rehearsal dataset into Python, run a standard principal-component analysis, and save the first two component vectors. They define a 2×D matrix. That matrix never changes; you’ll multiply each new latent vector by it to get (x, y) screen coordinates.
• If you prefer the parametric projector, train a tiny neural net (two dense layers with tanh or ReLU) to learn a 2D embedding that preserves distances in the rehearsal data—basically a UMAP or t-SNE style objective but expressed as a feed-forward network. When it looks good, freeze its weights and export them. At runtime you’ll call that network once per latent vector to get its projection.

3. Feed it into a renderer. In your Rust engine or in a companion process, every control block you compute (x_t^), multiply by the PCA matrix (or run the small network), and you get a 2D point. Send those (x, y) points, along with auxiliary data like phase ψ, energy ‖x_t^‖, residual, and per-limb energies, to a small graphics loop running at display rate.

4. Draw the geometry. Use a GPU library such as `wgpu`, `glium`, or `nannou` if you stay in Rust, or an `egui` canvas if you want it lightweight. Plot the current point as a dot, give it a short fading trail to show history, color it by phase ψ, size it by energy, and add the “aura” I mentioned earlier whose brightness corresponds to contraction residual. Smooth the incoming coordinates over a few frames so the trail glides instead of jittering.

5. Freeze the map. Because the PCA or projector weights are fixed, a region that corresponded to “low-energy torso sway” yesterday will still occupy the same place tomorrow. That’s what “you can revisit next week and still recognize” means—the axes don’t drift as new data arrives.

You don’t need the projector to run on the audio thread; it can run in the control thread at 60–120 Hz. You just keep pushing the projected coordinates into a ring buffer that the renderer consumes. The renderer paints them on screen in sync with the beat clock from Ableton Link or your own scheduler, so the picture moves with the same rhythm as the music.

If you build it this way, what you get is not a random cloud but a stable, interpretable 2D landscape of your own movements and phrases. Over time you’ll learn its geography: clusters where certain gestures or moods live, paths that correspond to transitions, regions that you visit during particular drops. That’s the “visual grammar” I was pointing to—the way a mathematical projection of latent vectors becomes a living map of how you play.

What I was describing there is the recipe for building the visualization layer that sits on top of your real-time engine—the part that lets you actually see Echelon’s latent manifold breathing in performance. It’s not something mystical; it’s a practical graphics pipeline that you can build the same way you’d build a lightweight visualizer or performance meter.

You start with a continuous stream of latent vectors (x_t^*) coming out of your solver every few milliseconds. Each of those vectors has, say, 32 or 64 dimensions—far too many to plot directly. The job of the projector (f_\phi) or the PCA basis is to compress those vectors into two coordinates that can be drawn on screen, and to do so in a way that doesn’t change its mind from one session to the next.

Here’s what that looks like in practice:

1. Gather rehearsal data. Record a few minutes of live motion-to-latent data while you perform. Store all the (x_t^*) vectors.

2. Compute a stable projection.

• If you go with the PCA approach, load that rehearsal dataset into Python, run a standard principal-component analysis, and save the first two component vectors. They define a 2×D matrix. That matrix never changes; you’ll multiply each new latent vector by it to get (x, y) screen coordinates.
• If you prefer the parametric projector, train a tiny neural net (two dense layers with tanh or ReLU) to learn a 2D embedding that preserves distances in the rehearsal data—basically a UMAP or t-SNE style objective but expressed as a feed-forward network. When it looks good, freeze its weights and export them. At runtime you’ll call that network once per latent vector to get its projection.

3. Feed it into a renderer. In your Rust engine or in a companion process, every control block you compute (x_t^), multiply by the PCA matrix (or run the small network), and you get a 2D point. Send those (x, y) points, along with auxiliary data like phase ψ, energy ‖x_t^‖, residual, and per-limb energies, to a small graphics loop running at display rate.

4. Draw the geometry. Use a GPU library such as `wgpu`, `glium`, or `nannou` if you stay in Rust, or an `egui` canvas if you want it lightweight. Plot the current point as a dot, give it a short fading trail to show history, color it by phase ψ, size it by energy, and add the “aura” I mentioned earlier whose brightness corresponds to contraction residual. Smooth the incoming coordinates over a few frames so the trail glides instead of jittering.

5. Freeze the map. Because the PCA or projector weights are fixed, a region that corresponded to “low-energy torso sway” yesterday will still occupy the same place tomorrow. That’s what “you can revisit next week and still recognize” means—the axes don’t drift as new data arrives.

You don’t need the projector to run on the audio thread; it can run in the control thread at 60–120 Hz. You just keep pushing the projected coordinates into a ring buffer that the renderer consumes. The renderer paints them on screen in sync with the beat clock from Ableton Link or your own scheduler, so the picture moves with the same rhythm as the music.

If you build it this way, what you get is not a random cloud but a stable, interpretable 2D landscape of your own movements and phrases. Over time you’ll learn its geography: clusters where certain gestures or moods live, paths that correspond to transitions, regions that you visit during particular drops. That’s the “visual grammar” I was pointing to—the way a mathematical projection of latent vectors becomes a living map of how you play.

echelon/
├─ Cargo.toml
├─ README.md
├─ LICENSE
├─ .cargo/
│  └─ config.toml
├─ xtask/
│  ├─ Cargo.toml
│  └─ src/…
│
├─ crates/
│  ├─ control-bus/
│  │  └─ src/lib.rs             # extend with FastState, SlowState, Telemetry, BeatClock
│  │
│  ├─ audio-engine/
│  │  └─ src/…
│  ├─ scheduler/
│  │  └─ src/…
│  ├─ ui-shell/
│  │  └─ src/
│  │     ├─ lib.rs
│  │     ├─ main_view.rs
│  │     ├─ mapping_view.rs
│  │     ├─ phrase_browser.rs
│  │     ├─ theme.rs
│  │     ├─ ui_to_scheduler.rs
│  │     └─ viz_panel.rs        # egui panel embedding viz (optional)
│  │
│  ├─ media/
│  │  └─ src/…
│  ├─ midi-osc/
│  │  └─ src/…
│  ├─ dsp-utils/
│  │  └─ src/…
│  ├─ link-clock/
│  │  └─ src/…
│  ├─ ts-shift/
│  │  └─ src/…
│  ├─ shmem-ipc/
│  │  └─ src/lib.rs
│  │
│  └─ viz/                      # new: latent-manifold visualizer as a library crate
│     ├─ Cargo.toml
│     └─ src/
│        ├─ lib.rs              # pub VizRunner/VizWidget API
│        ├─ projector/
│        │  ├─ mod.rs
│        │  ├─ pca.rs           # fixed 2×D matrix projection (loaded from assets)
│        │  └─ parametric.rs    # optional tiny MLP projector (feature "nn")
│        ├─ view/
│        │  ├─ mod.rs
│        │  ├─ manifold.rs      # 2D manifold + comet trail + contraction glow
│        │  ├─ phase_wheel.rs   # beat-phase ring overlay
│        │  ├─ satellites.rs    # limb-energy satellites
│        │  └─ atlas.rs         # rehearsal density map
│        ├─ runtime/
│        │  ├─ mod.rs
│        │  ├─ renderer.rs      # wgpu surface, frame loop (standalone window)
│        │  └─ egui_widget.rs   # egui Widget for embedding in ui-shell
│        ├─ data/
│        │  ├─ mod.rs
│        │  ├─ buffers.rs       # SPSC readers; backpressure = drop-old
│        │  └─ types.rs         # ProjectionSample { xy, psi, energy, residual, limbs[] }
│        ├─ clock.rs            # BeatClock helper (can subscribe to link-clock feed)
│        └─ config.rs           # VizConfig { projector, smoothing, colors, assets }
│
└─ apps/
   └─ studio/
      ├─ Cargo.toml
      └─ src/
         ├─ main.rs
         ├─ threads/
         │  ├─ audio_thread.rs
         │  ├─ scheduler.rs
         │  ├─ ui_thread.rs
         │  ├─ link_thread.rs
         │  └─ viz_thread.rs    # new: spins a window or mounts viz inside ui-shell
         ├─ config/
         │  ├─ runtime.yml
         │  ├─ midi.yml
         │  ├─ link.yml
         │  ├─ crates.yml
         │  ├─ ui.yml
         │  └─ viz.yml          # projector path, colors, smoothing, layers
         └─ logs/
            ├─ telemetry.log
            ├─ xruns.log
            └─ session.parquet

apps/
└─ viz-demo/                     # optional: stand-alone viz app
   ├─ Cargo.toml
   └─ src/main.rs                # reads a log or subscribes to rings; opens a viz window

Within this structure the data path is simple and safe. The scheduler already publishes BeatClock and an EngineShadowState; extend control-bus just enough to include FastState { x_star, psi, limb_energy, energy, residual }, SlowState { y_star, phi }, and Telemetry { cpu, xruns, callback_ns }. The control thread writes a decimated copy of those snapshots into a dedicated SPSC ring that the viz crate reads; the decimation rate should be tied to your UI refresh, not audio rate, so you never back-pressure the engine. The audio thread never writes to that ring and never reads from it, so it cannot be stalled by the renderer.

You have two integration modes and can support both behind a cargo feature. The first is embedded, where ui-shell depends on viz and mounts its egui_widget::ManifoldWidget as a tab or panel. In this mode the UI thread calls widget.draw(&mut egui::Ui, &mut viz_handle) every frame; the widget pulls the latest samples from the ring and renders into the existing egui surface. The second is standalone, where Studio spawns viz_thread.rs and calls VizRunner::spawn(rx_fast, rx_slow, rx_clock, rx_telemetry, VizConfig); the runner opens its own wgpu window and renders at monitor refresh, entirely decoupled from the rest of the UI. This second mode is useful during rehearsal or profiling because you can close the viz without touching the main UI stack, and if the GPU driver has an issue, it doesn’t affect the engine.

You do not have to move anything else to accommodate this. The engine continues to publish meter frames to ui-shell; the scheduler continues to consume UI intents and compile control events into the ring for the audio engine; the MIDI crate continues to map controllers. The viz crate is a consumer only. It reads from the same broadcast the UI reads, and it writes nothing back.

From the workspace’s point of view you only add one line to Cargo.toml to register the new crate, and, if you want to keep it optional, a viz feature in the Studio app that toggles the dependency so you can ship headless builds for servers or tests. A minimal public surface for the library looks like this

// crates/viz/src/lib.rs
pub struct VizConfig { / projector path, colors, smoothing, layers / }
pub struct VizRunner { / windowed renderer / }
pub struct VizWidget { / egui widget handle / }

impl VizRunner {
pub fn spawn(
cfg: VizConfig,
fast_rx: control_bus::RingReader<control_bus::FastState>,
slow_rx: control_bus::RingReader<control_bus::SlowState>,
clock_rx: control_bus::RingReader<control_bus::BeatClock>,
telem_rx: control_bus::RingReader<control_bus::Telemetry>,
) -> std::thread::JoinHandle<()> { / ... / }
}

impl VizWidget {
pub fn new(cfg: VizConfig, / same readers as above /) -> Self { / ... / }
pub fn ui(&mut self, egui_ctx: &egui::Context) { / draw manifold + overlays / }
}

This keeps the roles crisp. The projector lives inside crates/viz so it can be swapped between PCA and a tiny parametric net without anyone else noticing; the renderer is GPU-backed but isolated; the scheduler and engine do not change code paths in the audio callback; the UI gains a tab when you want it. If you ever decide to publish the visualization as open source while keeping the engine proprietary, having it as a separate crate from day one makes that trivial.

So the short answer is that the viz is not a new app; it is a library you drop under crates/ and call from the Studio binary or the UI. It reads from the same control-bus types you already defined, honors the same beat clock the scheduler already broadcasts, and paints the same geometry we discussed, without touching the render thread. It lives along the spine you’ve built, it never competes for the beat, and it makes the system’s inner motion visible with the same discipline that makes the sound unbreakable.