Failure Atlas
The hidden structure of forecasting mistakes.
Even after their ranges are corrected, every pair of weather ensembles misses together two to three times more often than chance. Whether that history predicts the next miss is still untested.
line width = how many times more often the pair missed together than if they missed independently · tap a link
2026-09-12 to 2026-10-17 · corrected 90% ranges · Offdiagonal failure-atlas-v1/weather/weather-ensemble-residuals.parquet · sha256 394c21cf…
00 · the matrix
Every pair, side by side.
Two systems missing together, compared with chance.
ECMWF × GEM
2.44×
Both missed 2.5% of the time; independent systems would both miss 1.0%. 90% interval 2.25× to 2.63×, from 984 place-days.
- history, not a warning
- corrected ranges
Each cell: how often both systems missed, divided by what independent misses would give. 1× = independent.
01 · after widening
Which shared misses are real?
Widen every range using only its own past errors, then count again.
COVID-19
2,199 outcomes · 23 wk
28 → 16 · 57% stay
RSV
529 outcomes · 10 wk
24 → 10 · 42% stay
Ensemble weather
15,314 outcomes · 2 wk
857 → 30 · 4% stay
Weather's shared misses were mostly ranges that were too narrow. In health, about half survive: real confidently-wrong consensus, still too rare to train on.
02 · weeks of evidence
Six weeks, counted by place-day.
9 / 32
W37
119 / 196
W38
122 / 196
W39
141 / 196
W40
134 / 196
W41
99 / 180
W42
12 Sep – 17 Oct 2026 · weekly place-days · corpus-v1-20261020 · range-history-v1 · manifest 624a4aa2
Evidence +−
Historical dependence measured
Yes. After fixing too-tight ranges, ensemble systems still miss together far more often than chance would allow.
Case-level risk not yet established
Untestable for now: once ranges are corrected, no group failures remain in the test weeks.
Decision value not tested
Not tested. Needs warnings posted before outcomes and a real decision to score.
Confidence spread used as disagreement
Fixed: disagreement now means how far the forecasts themselves sit apart.
Dependence measured once, for all time
Fixed: who-misses-together history is rebuilt from earlier weeks only.
Each forecast row counted as evidence
Fixed: one place-day or question counts once.
Ranges too tight to begin with
Found later: the 90% ranges held the outcome about 65% of the time. Repeated on corrected ranges (v0.77 step 1), the test window has zero group failures, so the 35% → 45% readout cannot be confirmed.
A leak in the averages (found 19 Oct)
Two average rates in v0.77 included future answers. Re-checked as v0.77r: the simple averaged who-misses-together number now helps on both weather sources, ForecastBench and ECB. v0.78 tests real pair histories instead of an average.
later test weeks · ~12,900 outcomes per row · fit before 2 Oct · corpus-v1-20261020 · range-history-v1
Single-run weather: no group-failure test events yet.
Waiting for Offdiagonal's single-run catch-up.
ForecastBench: too few questions answered in time.
Daily open-question files will allow warnings before outcomes.