SEA Phase 0 — MiniMax Scoring Latency Benchmark Results

SEA Phase 0 — MiniMax Scoring Latency Benchmark Results

Date: 2026-02-17 23:30 (re-run with live endpoint)
Endpoint: `http://localhost:18080`
Model: MiniMax-M2.5 (229B params, TQ1_0 GGUF quantization, 55GB)
Server: llama.cpp (llamacpp)
Prompt Template: Tier 2 (Skill Activation Judge)
Benchmark Script: `benchmark_minimax_scoring.py`

> Note: The originally spec'd MiniMax-3B-v0.1 has been superseded by MiniMax-M2.5,
> a 229B parameter reasoning model. This is significantly more capable but slower than
> a 3B model would be. The benchmarks below reflect the actual available hardware.

Endpoint Status: ONLINE ✅

1. Model Details

2. Benchmark Run 1: Token Generation Speed (150 max_tokens)

20 calls with the full SEA scoring prompt template, capped at 150 tokens to measure raw generation throughput.

Note: At 150 max_tokens the model's reasoning chain is truncated before producing
the JSON output. This run measures raw throughput, not end-to-end scoring.

3. Benchmark Run 2: Production Scoring (500 max_tokens, full output)

10 calls with the production-optimized scoring prompt. Model reasons through the
problem and produces a clean JSON scoring object.

Sample Scoring Output

{"score": 0.92, "reason": "The message asks about data accuracy and source trustworthiness, directly aligning with fact-checking and truth pursuit in decision-making contexts.", "inject": true}

Scoring Quality

The model produces excellent discrimination between relevant and irrelevant messages.
Scores are well-calibrated: philosophical questions score >0.85 for phi:veritas,
while "fix the typo" and "npm install" score <0.3.

Per-Call Detail

Observation: Low-relevance messages take longer (~6s) because the model reasons
more extensively before concluding the skill shouldn't activate. High-relevance
messages are faster (~2-3s) as the model quickly identifies the match.

4. Implementation Notes

### Reasoning Model Behavior
MiniMax-M2.5 is a reasoning model — it outputs `reasoning_content` (chain-of-thought)
before the final `content` (the JSON score). This means:
- Token budget must account for reasoning tokens (typically 150-300 tokens of reasoning + 50 tokens of JSON)
- `max_tokens: 500` is sufficient for full reasoning + output
- The JSON output sometimes wraps in markdown code blocks (`` ```json ```) — parser must handle both raw JSON and fenced JSON

### Production Prompt Optimization
The compact scoring prompt (used in Run 2) performs equally well as the full template:

You are {skill}'s activation judge. Decide if this skill should contribute.
SKILL: {desc}  HOT: {hot}  COLD: {cold}
MESSAGE: "{msg}"
CONTEXT: [last 3 exchanges]
Output ONLY JSON: {"score": 0.0-1.0, "reason": "one sentence", "inject": true/false}

Shorter prompts reduce prompt processing time (~154ms vs ~192ms).

5. Viability Assessment

Single Call Target: < 5s

P95 exceeds the 5s target. However, the P95 cases are all low-relevance messages
where the model reasons longer. In practice, these are exactly the messages where
scoring latency doesn't matter (no injection will be generated).

Full Pipeline Estimate (Parallel Scoring)

30-Second Injection SLO

Go/No-Go Matrix

Overall: GO — MiniMax-M2.5 scoring latency is viable for async Tier 2 use.
Mean single-call latency (3.8s) and P50 full pipeline (3.7s) both well within the
30-second injection delivery SLO. Scoring quality is exceptional with near-perfect
discrimination between relevant and irrelevant messages.

Recommendations for Phase 1

1. Set `max_tokens: 500` for scoring calls to allow full reasoning + JSON output
2. Parse JSON flexibly — handle both raw `{"score":...}` and fenced `` ```json...\n``` ``
3. Tier 2 is async — 3-6s latency is acceptable; user already has the main response
4. Consider reducing to 3B model if MiniMax-M2.5 proves too slow under load — a 3B model would likely achieve <500ms per call
5. Pre-warm the model with a startup scoring call to avoid cold-start penalties

Fallback Architecture (retained from initial assessment)

The multi-tier fallback remains valid even with MiniMax available:

                    ┌──────────────────┐
  User Message ───▶ │ Tier 1: Embed    │ < 50ms
                    │ (all 13 skills)  │
                    └────────┬─────────┘
                             │ top-K candidates (score > 0.4)
                    ┌────────▼─────────┐
                    │ Tier 2: MiniMax  │ ~3.5s (parallel)
                    │ (3-5 candidates) │
                    └────────┬─────────┘
                             │ score > 0.7
                    ┌────────▼─────────┐
                    │ Inject Skill     │
                    │ Perspectives     │
                    └──────────────────┘

  Fallback: If Tier 2 unavailable, Tier 1 scores
  directly determine activation (threshold 0.75).
  If Tier 1 unavailable, Tier 3 regex triggers.

Revised Latency Budget

---

SEA Phase 0 — Mac4 Ollama Embedding Verification

Date: 2026-02-17
Host: Mac4 ([ip])
Ollama Version: 0.16.2
Model: all-minilm (sentence-transformers/all-MiniLM-L6-v2)

---

1. Ollama Status

Note: `ollama` is not in the default SSH PATH. Use full path or access via HTTP API.

2. Model Installation

all-minilm was NOT pre-installed — only `llama3.2:3b` (2.0 GB) was present.

/Applications/Ollama.app/Contents/Resources/ollama pull all-minilm
# Downloaded: 45 MB model weights + 11 KB template + metadata
# Time: ~5 seconds

Models Now Available on Mac4

3. Embedding Test

curl http://[ip]:11434/api/embeddings \
  -d '{"model":"all-minilm","prompt":"philosophical truth"}'

Response (truncated)

{
  "embedding": [
    -0.005712718702852726,
    0.06258934736251831,
    -0.07911590486764908,
    0.05943027883768082,
    -0.023529276251792908,
    "... (384 dimensions total) ...",
    -0.01641690358519554,
    0.08414340019226074,
    0.03538107872009277,
    0.05721861124038696,
    -0.08861380070447922
  ]
}

4. Latency Benchmarks

Localhost (on Mac4, `time.perf_counter`)

Target: <10ms — PASS (7.45ms avg)

### Cold Start
- First call after model load: ~651ms (model loading into GPU/memory)
- Subsequent calls: <10ms consistently

### Network (from Mac1 via Tailscale)
- Round-trip including network hop: ~45ms
- Acceptable for cross-machine use; localhost recommended for latency-critical paths

5. Summary & Go/No-Go

Overall: GO — Ollama embedding infrastructure on Mac4 is verified and ready for SEA Phase 1. The all-minilm model returns 384-dimensional vectors in under 10ms (warm). Recommend implementing a keep-alive ping to avoid cold start penalties.