Paper 4 Research Closeout - 2026-05-03

Paper 4 Research Closeout - 2026-05-03

Final Decision

Paid Vast training is paused indefinitely. The remaining Vast instance was destroyed on
2026-05-03 to stop storage billing. `vastai show instances` was empty immediately after
destroying instance `35911134`.

The recurring heartbeat monitor `monitor-nko-anchor-audit` was also deleted because the
active remote audit no longer exists.

What Is Preserved Locally

The locally preserved anchor result is:

• Path: `[home]/Desktop/nko-brain-scanner/local_results_cache/paper4_reproduction_35205256/results.json`
• Script: `nko`
• Mode: `trajectory`
• TAR: `false`
• TTT: `false`
• LR: `0.0003`
• Batch size: `32`
• Dropout: `0.1`
• Seed: `42`
• Split: `232476 / 29060 / 29060`
• Best validation loss: `0.6358872798606507`
• Epochs trained: `47`
• Test CER: `0.2057`

This is the only locally retained result that directly supports the archived `20.57
N'Ko trajectory anchor.

Non-Comparable Runs

The April 22 safe matrix is preserved locally, but it is not a strict validation of the
20.57 anchor because it used `lr=0.0001`, not `lr=0.0003`.

Local April 22 safe matrix results:

• `nko_baseline_290596`: CER `0.3138`, best val `0.9559760437284497`, epochs `26`
• `nko_tar_290596`: CER `0.3169`, best val `0.9883851678457984`, epochs `19`
• `nko_trajectory_ttt_290596`: CER `0.3112`, best val `0.9368296321338029`, epochs `30`
• `latin_baseline_290596`: CER `0.3166`, best val `0.9543151013528554`, epochs `23`
• `latin_trajectory_290596`: CER `0.3281`, best val `1.0046055740518014`, epochs `22`

These runs are useful as exploratory evidence that the later setup behaved differently,
but they should not be used to refute or validate the `20.57

Strict Anchor Audit Attempt

The May 2 strict N'Ko audit was launched to test the suspected gotchas:

• Pair hash: `d67db91175be46e81655ec82a1b2576ba79a9e988fc6a0d2c3796c2e5a183cda`
• N'Ko vocab hash: `e3ab620c9d2f971603d76f953f2be40bf9283dfd99d6428c7d51a9a73246ea67`
• Split: `232476 / 29060 / 29060`
• Plain run: `nko_trajectory_290596`, `lr=0.0003`, `patience=8`
• TTT run: `nko_trajectory_ttt_290596`, `lr=0.0003`, `patience=8`

Last verified live state before Vast compute stopped:

• Plain strict N'Ko trajectory: around epoch `25`, step `5800/7265`
• Strict N'Ko trajectory+TTT: around epoch `18`, step `1100/7265`
• Both GPUs were active
• No final `results.json` existed yet

The audit did not complete, so it produced no final CER and cannot validate or
contradict the archived `20.57

Gotchas Found

- The earlier five-run matrix omitted the plain `nko_trajectory_290596` run needed to
compare directly against the archived anchor.
- The April 22 safe matrix used `lr=0.0001`; the archived anchor used `lr=0.0003`.
- A later N'Ko vocab path was contaminated by non-N'Ko/Latin/noise symbols. The strict
anchor vocab is the N'Ko block `U+07C0..U+07FF` plus space, hash
`e3ab620c9d2f971603d76f953f2be40bf9283dfd99d6428c7d51a9a73246ea67`.
- Hugging Face feature tensors are mixed shape; the patched loader normalizes
singleton-batch tensors such as `(1, 1500, 1280)` into `(1500, 1280)`.
- Vast stopped the last audit before completion, and the account ran out of credit.

Current Research Position

Use the archived `20.57
the current local evidence.

Do not claim the strict May 2 audit reproduced, invalidated, or improved the anchor.
It was correctly configured but incomplete.

Do not claim TAR or TTT conclusions from the April 22 safe matrix as final Paper 4
evidence, because the learning rate and audit conditions were not anchor-matched.

If This Is Resumed Later

Start from the strict audit scripts:

• `[home]/Desktop/nko-brain-scanner/docs/handoffs/train_vastai_tar_ttt_anchor_audit_20260502.py`
• `[home]/Desktop/nko-brain-scanner/docs/handoffs/vast-anchor-audit-strict-nko-2026-05-02.sh`

Before spending more money, require:

- Sufficient account credit for the full run, not partial progress
- An on-demand host with enough disk for the HF feature cache
- Explicit artifact download before destroying any instance
- A small checkpoint-sync path so partial training progress is not stranded on a Vast
stopped volume again