evidence
What has been shown, and what has not
This page is the audit trail. It reads a SurpriseReport field by field, explains how the probabilities and the abstentions are produced, states exactly which domains cleared the pre-declared gate in v0.6.0 and why the others did not, and names the gap v0.7.0 is being trained to close.
the output
A SurpriseReport, read line by line
Every scored entity comes back in the same typed contract. Nothing below is a summary statistic — each field is a separate commitment the model makes and can be wrong about.
example
entity gauge-11447650 divergence structure · cadence break validity reference holds ≤ 6h blindness 2 of 4 channels unobserved confidence 0.81 (calibrated) action request information
- entity gauge-11447650
- One scored unit of the stream — a gauge, a ticker, a station, a substation. Every entity keeps its own normalisation statistics and its own regime memory, so a quiet gauge and a flashy one are not judged on the same scale.
- divergence structure · cadence break
- The typed answer. The arrival pattern has changed shape rather than simply exceeding a level, which is why a threshold monitor on the same series would still be reporting normal.
- validity reference holds ≤ 6h
- A statement about the reference model, not about the world: beyond roughly six hours the predictor you rely on is expected to be outside the regime it was fitted in. This is the field that makes the output actionable for a scheduler.
- blindness 2 of 4 channels unobserved
- Missing inputs are carried as an explicit channel and never imputed. Low risk and low visibility are different states, and conflating them is how monitors get trusted at exactly the wrong moment.
- confidence 0.81 (calibrated)
- A probability fitted on the validation split, so 0.81 is intended to mean eighty-one out of a hundred. When the calibrated probability sits inside the abstention band, this field reads abstain instead of a number.
- action request information
- The decision the report commits to. Requesting information is a first-class outcome: the model is saying the cheapest next move is to close the coverage gap, not to act or to stand down.
calibration
What a stated probability is worth
Assurance requires numbers that mean what they say. Calibration is fitted on held-out validation data, measured on test, and gated.
what a probability means
Raw head outputs are not probabilities. Each head is Platt-scaled on the validation split — never on test — so a reported 0.81 is intended to mean that eighty-one out of a hundred comparable situations turn out to be transitions. The scaling parameters ship inside the released artifact and are baked into the ONNX graph, so the browser cannot silently disagree with the published numbers.
how it is measured
Expected calibration error bins predictions by stated confidence and measures the gap between the stated rate and the observed rate in each bin. The gate rejects a domain above 0.10 ECE even when its ranking is excellent, because a well-ordered but overconfident score cannot be used to size a response.
why ranking is not enough
Discrimination and calibration fail independently. Wildfire is the live example: the ranking carries signal, but its calibration is the weakest of the real corpora, so it stays a candidate rather than a promoted domain. Reporting AUROC alone would have hidden that.
abstention
When the model declines to score
Abstention is not a failure mode bolted on afterwards; it is one of the outputs the contract promises.
abstain is an answer
When the calibrated probability falls inside the abstention band, the report returns 'not enough evidence' instead of a number, and the recommended action becomes request-information. A monitor that must always answer is a monitor that is confidently wrong exactly when the world moves.
how the operating point is set
The band is chosen on the validation split at the point where declining to score buys the largest accuracy gain per unit of coverage lost. It is fixed before test scoring, like every other threshold in the protocol.
what it costs
The trade-off is published per domain rather than tuned per screenshot: coverage is the share of entities actually scored, and abstention gain is how much selective accuracy improves over scoring everything. Both appear in the per-domain cards below.
coverage of the evaluation
What v0.6.0 was actually measured on
Chronological splits with a horizon buffer between train and test, entity-level bootstrap confidence intervals, thresholds locked before the run, and a synthetic chaos corpus kept as a control that can never count as evidence about real data.
loading evaluation…
| domain | verdict | why |
|---|---|---|
| hydrology | promoted | Cleared the pre-declared AUROC floor and the 0.10 ECE ceiling on at least one target, with confidence intervals excluding chance. |
| markets | promoted | Cleared the pre-declared AUROC floor and the 0.10 ECE ceiling on at least one target, with confidence intervals excluding chance. |
| climate | promoted | Cleared the pre-declared AUROC floor and the 0.10 ECE ceiling on at least one target, with confidence intervals excluding chance. |
| seismic | not promoted · regression | four of five heads collapsed to a constant 0.500 while a logistic regression on the same features reaches 0.887 — rare-positive training defect, not degenerate labels; regression against v0.5.0 |
| spaceweather | not promoted · regression | four heads collapsed to a constant 0.500 against a 0.78–0.84 logistic baseline; regression against v0.5.0 |
| macro | not promoted | below the pre-declared AUROC floor on every target |
| wildfire | not promoted | below the pre-declared AUROC floor; calibration still the weakest of the real corpora |
| chaos | not promoted | synthetic corpus — never evidence about real data |
limits
What this evidence does not cover
- No promoted domain yet beats a plain logistic regression fitted on the same features. Until that holds, the falsifiable claim is not demonstrated — it is only unrefuted on the controls.
- seismic and spaceweather regressed against v0.5.0: heads that previously carried signal collapsed to a constant 0.500 under the new normalisation. Pin revision v0.5.0 if you need those two domains.
- In-browser scoring still runs the v0.5.0 ONNX export, because v0.6.0 has no ONNX artifact yet. The site does not claim otherwise anywhere.
- Nothing here is a validated operational detector. No vertical is running in production, and every corpus is public and retrospective.
in flight
v0.7.0 — the gate it has to clear
Training since 2026-08-27 23:49 UTC on 1x NVIDIA L4 (Hugging Face Jobs), 14 epochs across 8 corpora, with gifteval and lotsa held out entirely as external evidence.
- must beat a logistic regression on the same features
- collapse guard blocks any head within 0.005 of chance
- targets with fewer than 25 test positives are excluded, not scored
A pass means at least two real-corpus domains beat their logistic baseline while staying calibrated, with no head sitting at chance. Anything else is published as a failed gate, the way v0.4.0 was — the release ladder keeps its failures visible so the comparison stays honest.
next
Run it on a stream you care about
The workspace scores your own CSV or a connected feed with the same contract described above, abstentions included.