Evidence
What is claimed, and what was actually measured
The honest summary first: the trained checkpoint fails the benchmark shipped with it. Everything below is either a link to that evidence or a statement of its limits.
claim NOT supported
Release
0.2.0
v0.2.0-trained
Checkpoint
c6a85149c344
synthetic pretrain only
Benchmark verdict
NOT supported
ac896a3a69dd
Real-data results
none
no validated external corpus yet
The claim under test
A learned representation of model-relative surprise contains information about future regime change and model failure that is not captured by predictive uncertainty alone.
- — On the current synthetic benchmark the surprise index is inverted and fires on every clean episode, so it fails the degeneracy guards.
- — A plain logistic model over the same signals is the strongest detector in the table. That is the number to beat, and it is not beaten.
- — The uploaded-data path in Ingest and Signals therefore does not use the checkpoint. It uses stated reference predictors and a stated policy.
What would change the verdict
In order of expected effect.
- 1. Fix label/horizon alignment in the training objective; the inversion is consistent with a sign bug.
- 2. Train on a real corpus with generator-side labels replaced by measured reference degradation.
- 3. Re-run SurpriseBench with locked thresholds and the same controls, and publish the loss if it loses.
- 4. Only then let the checkpoint touch the scoring path used on user data.
Model architecture
Encoder, surprise latent z^S, heads and update policy.
SurpriseBench
Locked-threshold bakeoff against controls and baselines.
Reference inventory
The frozen predictors and their disagreement matrix.
Credible tails
Counter-expectation trajectories on the demo scenarios.
Surprise memory
Past admitted events and their precedents.
Analyst reports
Narrative form of a single admission decision.