SEA-0.2-COMPLETE
| Metric | Value | Target | Verdict | |--------|-------|--------|---------| | Model available | MiniMax-M2.5 (229B, TQ1_0) | running | **PASS** | | Mean latency (production) | 3,834ms | <5,000ms | **PASS** | | P50 latency | 3,449ms | <5,000ms | **PASS** | | P95 latency | 6,381ms | <5,000ms | **FAIL** (low-relevance only) | | Scoring discrimination | 0.90 vs 0.10 | clear separation | **PASS** | | Full pipeline P50 | ~3.7s | <30s | **PASS** | | JSON output quality | valid, well-structured | parseable | **PASS** |
Full Public Reader
SEA-0.2-COMPLETE
## Summary
Benchmarked MiniMax scoring latency against the live MiniMax-M2.5 endpoint at localhost:18080. Ran 30 scoring calls (20 throughput + 10 production-realistic) using the actual SEA Tier 2 activation-judge prompt template across 4 skill profiles and 20 test messages. Mean latency is 3.8s (P50: 3.4s) per scoring call with 500 max_tokens. Scoring quality is exceptional — 0.90 mean for relevant messages vs 0.10 for irrelevant. Full pipeline estimate (embedding + scoring + delivery) is ~3.7s P50, well within the 30s SLO. Verdict: GO.
## Changes
- File: `phase0-results.md` — replaced "UNREACHABLE" placeholder with full benchmark data (model details, 2 benchmark runs, scoring quality, viability assessment, go/no-go matrix, revised latency budget)
- File: `benchmark_minimax_scoring.py` — created benchmark script (20-call latency test with SEA scoring prompt template)
- File: `benchmark_results.json` — raw benchmark data (all 20 runs with per-call latency, token counts, server timings)
- File: `SEA-0.2-COMPLETE.md` — this completion report
Key Findings
| Metric | Value | Target | Verdict |
|---|---|---|---|
| Model available | MiniMax-M2.5 (229B, TQ1_0) | running | PASS |
| Mean latency (production) | 3,834ms | <5,000ms | PASS |
| P50 latency | 3,449ms | <5,000ms | PASS |
| P95 latency | 6,381ms | <5,000ms | FAIL (low-relevance only) |
| Scoring discrimination | 0.90 vs 0.10 | clear separation | PASS |
| Full pipeline P50 | ~3.7s | <30s | PASS |
| JSON output quality | valid, well-structured | parseable | PASS |
### Important Discovery: Model Change
The endpoint serves MiniMax-M2.5 (229B params), not the originally spec'd MiniMax-3B-v0.1. This is a much more capable reasoning model with chain-of-thought output. Implications:
- Scoring quality is excellent (the model reasons about relevance before scoring)
- Latency is higher than a 3B model (~3.8s vs estimated ~200-500ms for 3B)
- For async Tier 2 use, this is acceptable — user already has the main response
- If latency becomes a concern, a 3B model could be added on a separate port
### Production Notes
1. `max_tokens: 500` needed for full reasoning + JSON output
2. JSON parser must handle markdown-fenced output (`` ```json {...} ``` ``)
3. Low-relevance messages take longer (~6s) but don't generate injections
4. Pre-warm recommended to avoid cold-start penalties
## RTD Verification
- [x] Structure: benchmark script, results JSON, and updated phase0-results.md all present
- [x] Compilation: Python script runs cleanly, produces valid JSON output
- [x] Integration: results documented in existing phase0-results.md alongside Mac4 section
- [x] Content: 30 scoring calls executed, P50/P95/min/max/mean computed, scoring quality validated
- [x] User Journey: benchmark can be re-run with `python3 benchmark_minimax_scoring.py`
- [x] Deployment: committed to sound-sigils-dep branch
## Cross-Pollination
N/A — no cross-track dependencies
Promotion Decision
Promote into a technical note or architecture paper with implementation anchors.
Source Anchor
skill-entity-architecture/SEA-0.2-COMPLETE.md
Detected Structure
Evaluation · Code Anchors · Architecture