surprisebench
The leaderboard, and the protocol behind it
A leaderboard is only worth as much as the protocol behind it. This is how episodes are split, how thresholds are locked, which controls run alongside every method, and what each baseline stands for.
held-out evaluation · surprisebench/1.0.0 · thresholds locked on validation
Every detector, identical inputs, identical labels — controls included
The primary column is AUPRC, read against a label prevalence of 0.399. A constant predictor and a uniform random predictor are scored in the same table, and every method is re-scored against shuffled labels; a method that does not clearly beat all three has not been shown to work. Lead time at a fixed 10% step false-positive rate is reported only when a method respects that operating point — an always-firing detector shows —, not a long warning.
| method | AUPRC | AUROC | lead@FAR | step FPR | detect | ECE | flags |
|---|---|---|---|---|---|---|---|
| SI — z^S failure head | 0.990 | 0.993 | 7.25 | 0.100 | 1.000 | 0.030 | |
| Logistic on hand-engineered signals | 0.894 | 0.921 | 4.10 | 0.069 | 0.649 | 0.042 | |
| Conformal width | 0.715 | 0.706 | 4.83 | 0.075 | 0.508 | 0.139 | |
| CUSUM change-point | 0.699 | 0.803 | 2.66 | 0.080 | 0.340 | 0.162 | |
| OOD score | 0.691 | 0.719 | 5.13 | 0.078 | 0.518 | 0.186 | |
| Anomaly score | 0.686 | 0.720 | 2.91 | 0.069 | 0.429 | 0.144 | |
| Disagreement × impact | 0.604 | 0.696 | 3.29 | 0.080 | 0.393 | 0.278 | |
| kNN novelty | 0.589 | 0.720 | 7.40 | 0.139 | 0.707 | 0.258 | |
| Control — uniform random | 0.399 | 0.500 | 2.99 | 0.099 | 0.220 | 0.260 | |
| Ensemble variance | 0.396 | 0.371 | 1.95 | 0.086 | 0.178 | 0.327 | inverted |
| Control — constant predictor | 0.394 | 0.500 | — | 1.000 | 1.000 | 0.101 | over-alarming · always-on · constant |
| Predictive entropy | 0.284 | 0.239 | 5.64 | 0.104 | 0.319 | 0.598 | inverted |
what each column measures, and why it matters+
Primary column. With rare breaks, precision–recall area degrades honestly where AUROC still looks flattering; read it against prevalence, which is the score a random detector gets.
Probability a break step outranks a clean one. Reported for comparability with the uncertainty literature, not as the decision metric.
Warning time is only worth anything at a fixed alarm budget, so a method that ignores the budget shows a dash instead of a long lead.
The realised cost of the alarm. It is what makes lead time comparable across rows rather than a race to alarm first.
Coverage in episode units — how often a real break is caught at all, which AUPRC can hide by ranking well inside a few episodes.
A score that ranks well but is systematically overconfident cannot be used as a threshold in an operating procedure.
what each method scores, and why it is in the table+
The failure head read off the learned surprise latent — a function of how reality relates to the whole reference set, not of any single predictor's confidence.
why it mattersThis is the subject of the claim. If it cannot beat every row below it on identical inputs and labels, the learned representation carries nothing the conventional signals do not already carry.
Supervised logistic regression on the hand-engineered signals, trained on the same labels.
why it mattersA strong supervised baseline, not a ceiling. It sees the answers during fitting, so it shows how much of the label a simple supervised model recovers from features alone — a better-specified supervised model could score higher.
Spread of the member predictions at each step.
why it mattersThe standard answer to 'are the models unsure?'. It measures disagreement about the next value, which is not the same as the references losing validity.
Entropy of the predictive distribution.
why it mattersClassic epistemic-uncertainty proxy. High entropy in a noisy but stable regime is not a break, which is exactly the confusion the claim is testing.
Mahalanobis distance of the current input from the training distribution.
why it mattersDetects inputs the model was never trained on. It is blind to breaks that occur inside the training support.
Standardised residual against the primary reference.
why it mattersThe default anomaly detector. It fires on one-off outliers as loudly as on structural change — the false_anomaly episodes exist to punish that.
Width of the conformal prediction interval at the locked coverage level.
why it mattersA calibrated uncertainty band. Widening intervals are the honest baseline for 'the model is losing grip'.
Cumulative-sum statistic over residuals with a drift allowance.
why it mattersThe textbook change-point detector, and the toughest classical rival on timing: it is designed to accumulate small persistent shifts.
Mean distance to the k nearest historical states.
why it mattersNon-parametric novelty. It answers 'have we been here before?' without any model of what should have happened here.
Ensemble disagreement weighted by the consequence model.
why it mattersThe strongest hand-built composite — the thing a good team would build without a learned latent. It is the honest 'do you even need a new model?' comparison.
A predictor that emits the same score at every step.
why it mattersAny metric a constant can score well on is a property of the label prevalence, not of detection.
Uniform random scores.
why it mattersChance floor. Every method is also re-scored against shuffled labels and must collapse to this level when it does.
A destructive control that removes any real association between score and label.
why it mattersIt is the guard against leakage and metric artefacts: a method must collapse to chance here or its headline number is not evidence.
Synthetic corpus, ten labelled mechanisms including recovery and false_anomaly episodes that contain no structural break. Disjoint train/validation/test seeds, thresholds locked on validation and applied unchanged to test, confidence intervals bootstrapped over whole episodes. These are properties of the harness, not evidence about the world.
The intervals estimate evaluation-sample uncertainty only — how far a number would move on another draw of test episodes from this generator. They do not cover training-seed variance, generator choice, or transfer to real data; every row comes from one training seed. And logistic_signals is a strong supervised baseline, not an upper bound: it sees the labels during fitting, but a better-specified supervised model on the same features could score higher.
Leave-one-mechanism-out
The in-distribution leaderboard cannot tell you whether z^S learned a general representation of model failure or just this generator’s vocabulary of failure. Each row below trains a fresh model with one failure mechanism withheld, then evaluates on exactly that unseen mechanism plus clean episodes, against the same baselines. Losses are shown, not only wins.
| held-out mechanism | z^S AUPRC | best baseline | margin | transfers |
|---|---|---|---|---|
| cross_model_conflict | 0.893 | anomaly_score 0.999 | -0.106 | no |
| support_drift | 0.800 | conformal_width 0.966 | -0.166 | no |
| constraint_tension | 0.597 | conformal_width 0.988 | -0.392 | no |
| sensor_blindness | 0.567 | ood_score 0.762 | -0.195 | no |
| class_failure | 0.692 | logistic_signals 0.991 | -0.299 | no |
| modality_loss | 0.610 | ood_score 0.770 | -0.160 | no |
| delayed_consequence | 0.794 | logistic_signals 0.789 | +0.006 | no |
Each row trains a fresh model with that failure mechanism withheld from the training corpus, then evaluates on episodes of exactly that mechanism plus clean episodes. Thresholds are locked on a validation split of the same held-out mechanism. Fewer epochs than the shipped checkpoint, so rows are comparable to each other, not to the in-distribution leaderboard.
protocol
The evaluation protocol, in full
The protocol
Disjoint corpora
The model never sees the episodes it is graded on.
Train, validation and test corpora are generated from disjoint seed ranges. Reported numbers come from the test split only; the validation split exists solely to lock thresholds.
Locked operating point
Thresholds are chosen before the test set is touched.
Every method is thresholded at the same false-alert rate on the validation split, then that threshold is frozen and applied to test. Reported operating point: FAR = 0.1. Without this, lead time can be bought by simply alarming more often.
Identical inputs, identical labels
The subject and its rivals see exactly the same data.
The surprise index and every baseline are computed from the same reference tensor over the same episodes with the same onset labels. No method gets a private feature or a private label definition.
Degeneracy controls
The harness can catch itself cheating.
A constant scorer and a random scorer are graded alongside the real methods, and every method is re-graded against shuffled labels. If a shuffled-label copy scores well, the metric — not the model — is the thing being measured.
Leave-one-mechanism-out
The only test that separates learning from memorising the generator.
A fresh model is trained with one failure mechanism entirely withheld, then evaluated on exactly that unseen mechanism. Seven mechanisms, seven retrained models, all rows published including losses.
The metrics
Plain reading on the left, the definition that makes it checkable on the right. Nothing here is reported alone: ranking metrics are always shown with prevalence, lead time always with its false-alert rate.
How well the score ranks the steps that really are pre-failure above the ones that are not, when those steps are rare.
Area under the precision–recall curve over per-step scores. The random-scorer floor is the prevalence of positive steps, not 0.5 — a number only means something next to the prevalence printed with it.
The chance a randomly chosen pre-failure step scores above a randomly chosen calm step.
Area under the ROC curve. Insensitive to prevalence, which is exactly why it is reported next to AUPRC and never instead of it. Below 0.5 means the score is inverted, and the harness flags it rather than silently flipping the sign.
How many steps of warning you actually get, at an alert rate you agreed to live with.
Mean number of steps between the first alert and the labelled onset, using the threshold locked on validation at the stated false-alert rate. Lead time quoted without a fixed FAR is uninterpretable.
How often it cries wolf when nothing is wrong.
Fraction of episodes containing no failure in which the method raises at least one alert at the locked threshold. The cost side of every lead-time claim.
Whether the numbers mean what they say, not just whether they rank correctly.
Expected calibration error and Brier score on the per-horizon failure head. A well-ranking, badly calibrated model is usable for triage and unusable for anything that consumes the probability.
If you can only investigate the top few alerts, how many are real?
Precision within the k highest-scoring steps, with k set to the number of true positives. The operational read of the ranking.
How much of the gap could be luck of the draw in which episodes landed in the test set.
95% percentile intervals from resampling the evaluation episodes. These estimate evaluation-sample uncertainty only. They say nothing about variance across training seeds, initialisations, or corpus regenerations, and must not be read as such.
What each baseline represents
Each rival stands for a different existing answer to “is something wrong?”. The claim is not that they are bad detectors — several of them are excellent — but that they measure uncertainty about outcomes rather than validity of models. Beating them is the only thing that would make that distinction real.
| ensemble_variance | The references disagree with each other. | Spread of the reference predictions at each step. The classic proxy for epistemic uncertainty, and the most direct rival to the disagreement axis of z^S. |
| predictive_entropy | One reference says it is unsure. | Entropy of the reference predictive distribution. Measures a model's stated uncertainty about outcomes, not whether the model itself is still valid — the distinction the whole claim rests on. |
| ood_score | This input does not look like the training data. | Distance from the fitted support of the reference inputs. Detects unfamiliarity of inputs; says nothing about whether unfamiliar means consequential. |
| anomaly_score | This observation is weird. | Unsupervised outlier score on the observation stream. Strong on abrupt, high-amplitude breaks; blind to slow erosion of validity. |
| conformal_width | The prediction interval is getting wide. | Width of the conformal interval at the locked coverage level. A principled uncertainty quantifier and, on several held-out mechanisms, the hardest baseline to beat. |
| cusum_changepoint | The mean has shifted. | Classical cumulative-sum change-point statistic on the residual stream. The right rival for the regime-change part of the claim. |
| knn_novelty | No near neighbour in the reference history. | Distance to the k nearest historical states. Novelty as memory lookup rather than as learned structure. |
| disagreement_x_impact | Disagreement, weighted by how much it would matter. | A hand-built product of spread and consequence — the strongest baseline that a practitioner could assemble without training anything. |
| logistic_signals | A supervised model given the same signals and the labels. | Logistic regression fitted on the same features with access to the training labels. This is a strong supervised baseline, not a ceiling: it is not an upper bound on achievable performance, only a well-tuned conventional competitor. |
| control_random / control_constant | Deliberate nonsense. | A random scorer and a constant scorer. If either scores well on a metric, that metric is broken for this corpus and the finding is about the harness. |
| *__shuffled_labels | Every method, graded against nonsense labels. | The same pipeline with labels permuted. Near-floor results here are what licence any positive result above. |
In distribution, the surprise index leads the leaderboard. Out of distribution it does not: across 7 leave-one-mechanism-out runs it beats the best conventional baseline on 0 of them. On this corpus, the in-distribution margin is largely knowledge of the generator’s vocabulary of failure rather than a transferable representation of model failure. That is a negative result, it is published unchanged, and it is the thing the next training corpus has to fix.
walkthrough
One held-out example, scored step by step
Every number in the leaderboard comes from arithmetic on a sequence of scores. Step through a single test episode and watch AUPRC, AUROC, lead@FAR and ECE being computed on it.
One held-out episode
Sixteen steps of a single test episode. The reference set holds until step 9, where a slow regime shift begins (shaded). The solid line is the surprise index; the faint line is the strongest conventional rival on identical inputs. Everything below is computed from exactly these two sequences.
Illustrative episode drawn from the synthetic held-out split; the metrics above are recomputed live from the plotted scores, so they are exact for this episode and not the published aggregates.