Research · archive
Past training runs.
Finished and stopped runs, newest first. Nothing here is training now.
Last finished runSize ladder: 13.8M, 44.7M, 134.5M+
bigger vs better · v0.82.5 size ladder on a second H200 · best full-model development score
1.8M
v0.82.4 · comparison
not measured yet
13.8M
size S · judged
1.118
44.7M
size M · queued
not measured yet
134.5M
size L · queued
not measured yet
job 6adabe4c925af0e27d4f994f · bigger counts only where it scores lower on the same rivals.
energy: Zeno beats every rival. 12 fields keep the reference.
energy
validated improvement
18% lower loss than calibrated reference · 24 groups
relations: no pairs 1.00 · shuffled 1.17
crypto
calibrated reference
best on selection: track-record tree
relations: no pairs 0.99 · shuffled 0.99
economy-ecb
calibrated reference
best on selection: track-record tree
relations: no pairs 1.00 · shuffled 0.97
economy-philfed
calibrated reference
best on selection: calibrated reference
relations: no pairs 0.91 · shuffled 0.93
economy-philfed-point
calibrated reference
best on selection: calibrated reference
relations: no pairs 1.00 · shuffled 1.03
health-covid
calibrated reference
best on selection: Zeno
relations: no pairs 1.00 · shuffled 1.00
health-flu
calibrated reference
best on selection: Zeno
relations: no pairs 1.00 · shuffled 1.00
health-rsv
calibrated reference
best on selection: Zeno
relations: no pairs 0.98 · shuffled 1.05
markets
calibrated reference
best on selection: calibrated reference
relations: no pairs 1.00 · shuffled 1.00
markets-cross-venue
calibrated reference
best on selection: Zeno
relations: no pairs 0.73 · shuffled 0.79
sea
calibrated reference
best on selection: Zeno
relations: no pairs 0.96 · shuffled 0.94
weather-sit-analysis
calibrated reference
best on selection: Zeno
relations: no pairs 1.00 · shuffled 1.00
weather-sit-station
calibrated reference
best on selection: Zeno
relations: no pairs 0.99 · shuffled 1.05
air-pressure-msl
insufficient evidence
air-wind-10m
insufficient evidence
urban-heat
insufficient evidence
Relations ratio = loss without ÷ loss with · above 1.00 means relationships helped · below 1.00 means the model did better without them. Promotion eligible: no (needs fresh prospective cases).
Only rounds observed since you opened this page (refreshes every 30 s) · the line restarts when a new variant begins.
Readings from each job's own status file, kept only while this page is open.
ingredient test · which inputs matter
No finished variant yet. Dots appear as seeds finish.
Live run record unreadable right now — not the same as stopped. Retrying every 30 s.
- reads
- up to 16 forecasters per question: each one's forecast, track record and type
- relations
- who missed together before, shared data sources, same family — fed into attention itself
- world
- 32 measured conditions at decision time (plus which are missing)
- size
- v0.82.4: 3 layers · width 128 · 1.8M · v0.82.5 ladder: 13.8M / 44.7M / 134.5M
- outputs
- a corrected forecast + two warnings: one forecaster wrong, the whole panel wrong
- judged
- per field, per skill, against the reference, a tree model, track records and the controls
Public beta recordv0.80.2: the public beta and its verdict+
- Latest completed
- v1.07 · completed · not served · v1.08 must beat v1.06 case for case
- Last finished
- v0.82 · judged 22 Oct · energy +17.5% vs track records, +15.6% vs calibrated reference; sea +25% and crypto +23% vs calibrated reference · no skill passes alone (two rivals not measured) · published on Hugging Face
full · start 0
Reference = 1.00 · lower is better
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 4, stopped at 8
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 6, stopped at 10
Saved history · best at round 1, stopped at 5
Saved history · best at round 5, stopped at 9
Saved history · best at round 4, stopped at 8
Saved history · best at round 2, stopped at 6
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 1, stopped at 5
Saved history · best at round 3, stopped at 7
Saved history · best at round 4, stopped at 8
Campaign map · seven controlled variants
full
no history
no pairs
no identity
no world
no pretrain
lodo:health-flu
v0802-full-20261021T0615Z · measured 2026-10-21 08:11:16Z. Completed curves use saved round histories; the active curve contains only observed rounds, with gaps unfilled. Development trends are not a sealed verdict. Most starts peak within the first rounds and stop early after four rounds without improvement — a sign to investigate overfitting in the next version, not a crash. The final verdict comes from a corrected judge that scores the averaged model customers receive.
00 · verdict
Which fields earned a correction
Only the corrected judge counts: averaged forecasts, each variant on its own inputs, one test across every field.
- Energy demandvalidated improvementvalidated improvement
- Weather vs airportsvalidated improvementvalidated improvement
- Air pressurecalibrated referencecalibrated reference
- Wind 10 mcalibrated referencecalibrated reference
- Cryptocalibrated referencecalibrated reference
- ECB surveycalibrated referencecalibrated reference
- Phil. Fed (prob.)calibrated referencecalibrated reference
- Phil. Fed (numbers)calibrated referencecalibrated reference
- COVID-19calibrated referencecalibrated reference
- Flucalibrated referencecalibrated reference
- RSVcalibrated referencecalibrated reference
- Market pricescalibrated referencecalibrated reference
- Polymarket traderscalibrated referencecalibrated reference
- Seacalibrated referencecalibrated reference
- Space weathercalibrated referencecalibrated reference
- Single-run weathercalibrated referencecalibrated reference
- Weather vs analysiscalibrated referencevalidated improvement
- Agriculture raininsufficient evidenceinsufficient evidence
- Air quality PM2.5insufficient evidenceinsufficient evidence
- Cross-venue marketsinsufficient evidenceinsufficient evidence
- Urban heatinsufficient evidenceinsufficient evidence
Fixed scoring: a few zero-width standard ranges had scored losses in the billions. The re-check scores the same saved models with one shared minimum width learned from training data only. It looks again at periods already seen, so it is a correction, not new proof; a release needs a fresh period.
01 · field readings
Where the reference improved
Each dot is a variant's measured development loss from the finished run, not its final verdict.
Current variant only, not the full-model verdict. Teal is below the reference; ochre is above it. No field is promoted by these readings.
Run stages and frozen training data
- 01Data builtdone
- 02Time auditdone
- 03Short GPU testdone
- 04Owner approvaldone
- 05Full trainingdone
- 06Judgingdone
- 07Verdictdone
The approved full run has 7 variants × 3 seeds, up to 30 rounds each. Development numbers are not final evidence.
- forecasts
- 12,134,209
- panels
- 4,599,358
- with outcome
- 2,840,152
- distinct events
- 1,265,459
Time audit passed: 0 forecasts used after their decision time, 0 outcomes known before it. Most of the growth over v0.80 comes from the new air pressure and wind fields. Every field counts equally in judging, however large it is.
02 · the model
A small panel reader
It reads who forecast what, who tends to miss together, and the conditions, then decides whether to keep or correct the reference.
Reads the panel
Up to 16 forecasts at a time, each with its track record, shown as positions relative to the calibrated reference.
Compares forecasters
3 attention layers (4 heads) that weigh each forecaster against the others, using time-valid pair history: who misses together.
Sees conditions
32 decision-time world readings for weather cases; switched off when absent.
Answers four things
A bounded correction, a calibrated probability, the chance of a confident miss, and the chance of a shared failure.
Small on purpose: v0.69 tested larger models: bigger was worse on the locked rule. An untrained model gives exactly the reference, so any change from it has to be learned.
03 · how it is judged
One test for the whole campaign
Each field must beat the reference, a tree model and a track-record model on two periods, with the correction for testing many fields at once.
Passes
Corrected answers are served for that field, from whichever model won.
Even
The calibrated reference is served.
Too few outcomes
Insufficient evidence; the reference is served.
Transfer
Trained without a field, then scored on it. Reported per field, never averaged.
- sealed period = historical confirmation
- airport readings: assumed timing
- v0.57 stays served
Earlier recordv0.80 and v0.79e: superseded research+
v0.80 · superseded
What v0.80 found
Research candidates before the data repairs; retested in v0.80.2.
v0.79e · verdict
Reading the panel beats track records.
Research only; superseded by v0.80.
Sea
3,175 outcomes
45% vs 30%
beats all four referencesWeather · single-run
5,532 outcomes
48% vs 11%
beats all four referencesSpace weather
73 outcomes
14% vs 0%
beats all four referencesPrediction markets
2,012 outcomes
no panel test
beats all four referencesEconomy · Phil. Fed points
120 outcomes
0% vs 0%
beats all four referencesWeather · ensemble
15,797 outcomes
22% vs 11%
beats all four referencesRSV
796 outcomes
14% vs 10%
beats all four referencesCrypto
24,996 outcomes
no panel test
loses to the tree referenceEconomy · Phil. Fed ranges
34 outcomes
7% vs 0%
no reliable differenceCOVID-19
2,786 outcomes
10% vs 6%
no reliable differenceNot testable: Economy · ECB. Warnings = share of failed questions caught when reviewing the riskiest 10% of questions (random = 10%). 134 of 134 models passed the validity check.
v0.79e · finished
The 13-source panel reader
All 134 research models trained, saved and re-checked.
- ✓ Data rebuilt
- ✓ Timing measured
- ● Research training
- ○ Comparison
- ○ Confirmation
- ○ Serving
How it reads a panel
01 · Input
every forecaster's answer, native range, horizon, and time-valid track record (50+ earlier answers)
02 · Core
attention across the whole panel; pair history tested as a variant
03 · Heads
forecaster miss risk · confidently wrong consensus
04 · Checks
main 0.5M · 2M size check · no pair history · no attention · history only · 15% source hiding · leave one source out
- Architecture
- Panel reader · 0.5M and 2M parameters
- Data
- 13 sources · v079e-cache-20261020T0835Z-clockfix-all13
- Models
- 134 research models over 9 time folds
- Measured speed
- 2.8 s (0.5M) / 3.4 s (2M) per pass · worst case 3.5 h
- Scoring
- not scored yet · per-model numbers are diagnostic
- Answers users
- no · v0.57 stays served
Abilities, judged separately
Range calibration
measured separately (v0.79d)
Ensemble weather ranges 66% to 90% covered; COVID-19 85% to 91%. Not learned by v0.79e.
Forecaster miss risk
training now
How likely each confident forecaster is to miss, read from the whole panel.
Confidently wrong consensus
training now
Whether most independent families miss together. Not testable where outcomes are too few.
Unseen sources
experimental, unsupported
Leave-one-source-out tests decide; until then, new sources get no supported answer.
Which sources can test a confidently wrong consensus
| Source | Consensus failed | Judged | Status |
|---|---|---|---|
| ECB survey | — | 0 | not testable |
| Philadelphia Fed survey | 21.4% | 392 | testable |
| economy-philfed-point | 16.8% | 441 | testable |
| Crypto price ranges | — | 0 | not testable |
| COVID-19 hospitalisations | 12.1% | 1,281 | testable |
| Prediction markets | — | 0 | not testable |
| RSV | 13.0% | 323 | testable |
| ForecastBench | — | 0 | not testable |
| Space weather | 2.0% | 709 | testable |
| Sea conditions | 2.2% | 2,540 | testable |
| Deterministic weather | 5.7% | 5,408 | testable |
| Ensemble weather | 28.6% | 15,698 | testable |
- research only
- not scored yet
- v0.57 stays served
- unseen sources unsupported
v0.79e · curves
Every saved model, learning.
Read from each model's own saved record on Hugging Face.
selected model
full reader · 0.5M · start 1
best after pass 24 of 32 · 38.6s
tested from 2026-04-11
- full reader 3
- no pair history 5
- no attention 1
- track records only 0
- 2M size
saved and re-checked
73 / 134
periods started
5 of 9
invalid values
0
- curves are checking loss, not results
- scores wait for the comparison report
- v0.57 stays served
Each line is one saved research model: its checking loss after every pass, smoothed. Lower is better, so falling lines mean learning; a curve stops when checking stops improving.
- Serving
- v1.06
- Completed research
- v0.78t · neural campaign scored · not good enough to switch
- Scored research
- v0.79e · beats references in 7 of 10 sources · research only
- Previous run
- v0.80 · flu and energy candidates · others keep the reference
- Older beta
- v0.80.2 · finished · energy + weather vs airports passed · opt-in beta · fixed-scoring re-check adds weather vs analysis (retrospective) · others keep the reference
- Live test
- Forecast Failure Watch · every open weather case registered before its outcome · scored hourly
- Market test
- All-markets rerun on Offdiagonal v6.11 · 1,329 city-days, 15 cities · prices match Polymarket exactly on a 364-point check · the market beats the weather models in every period (later test: 0.047 vs 0.062 error, 90% range excludes no difference) · 401 city-days had no price before the decision and were left out
- Latest completed
- v1.07 · completed · not served · v1.08 must beat v1.06 case for case
- Last finished
- v0.82 · judged 22 Oct · energy +17.5% vs track records, +15.6% vs calibrated reference; sea +25% and crypto +23% vs calibrated reference · no skill passes alone (two rivals not measured) · published on Hugging Face
- Current state
- Zeno v1.06 serves Check, API and MCP: one network across 13 fields, with its own reference retained where an edge is unproven. The completed run and recovery passed exact replay. Against the previous model on identical cases and grading: crypto has 7.9% less error (95% range 4.9–10.4%), weather history is 1.0% worse and ECB is tied. v1.07 is checking new-data eligibility, not training; hurricane publication clocks and matched joint-warning evidence remain required. Live sealed-forecast performance is not yet established. · checked 8 Nov 2026
- Market prices
- Offdiagonal v6.11 frozen 22 Oct · all 19,030 markets done · 19,019 priced, 119,744 price points · 11 have no history on Polymarket (reasons recorded) · 0 failed, none zero-filled · fingerprints re-checked by Zeno
- v0.79 (stopped rerun, older data) neural campaign
- Evaluated · diagnostic scores on weeks already seen · diagnostic only
Full recordv0.79 and v0.78: curves, checks, results+
01 · v0.79, stopped
A quick rerun on older data
Sixteen models on the older data, stopped on 20 Oct. Its numbers are diagnostic only and are not v0.79e.
01 · Inputs
15 per forecaster: track record, own confidence, place in the panel, time of day and season
19,456 / 71,680 weights
02 · Pair memory
4 per pair, as of the decision: shared outcomes, joint misses, "B misses after A", same family
17,708 / 68,024 weights
03 · Panel encoder
3 attention layers across up to 16 forecasters; pair history steers who listens to whom
594,816 / 2,369,280 weights
04 · Two heads
Will this forecast be confidently wrong? Will ≥2 families miss together (only where 3+ families exist)?
784 / 1,552 weights
05 · Calibration
Per source, on a later stretch of time that never chose the checkpoint
01 · Inputs
15 per forecaster: track record, own confidence, place in the panel, time of day and season
19,456 / 71,680 weights
02 · Pair memory
4 per pair, as of the decision: shared outcomes, joint misses, "B misses after A", same family
17,708 / 68,024 weights
03 · Panel encoder
3 attention layers across up to 16 forecasters; pair history steers who listens to whom
594,816 / 2,369,280 weights
04 · Two heads
Will this forecast be confidently wrong? Will ≥2 families miss together (only where 3+ families exist)?
784 / 1,552 weights
05 · Calibration
Per source, on a later stretch of time that never chose the checkpoint
Totals: 632,764 (small) and 2,510,536 (large). Controls are matched within 5%.
| Version | Small · starts 0 1 2 | Large · starts 0 1 2 | ||||
|---|---|---|---|---|---|---|
| Full: pair history on | ||||||
| Pair history removed | ||||||
| Pair history shuffled (time-valid) | ||||||
| No attention, same size | ||||||
| Track records only | ||||||
16 of 16 slots verified · Evaluated · diagnostic scores on weeks already seen · update #1114
Filled = finished and healthy · ochre = finished but noisy or not learning (reported by the job itself).
02 · learning curves and checks
Is it actually learning?
Curves come from the run's own progress file; checks are not results.
0.5M-no_attention-s0
21 epochs · 2,247 steps · healthy · kept 0.6442 vs reference 0.7355
0.5M-no_attention-s1
28 epochs · 2,996 steps · healthy · kept 0.6382 vs reference 0.7355
0.5M-no_attention-s2
40 epochs · 4,280 steps · healthy · kept 0.6308 vs reference 0.7355
0.5M-no_attention-s3
60 epochs · 6,420 steps · healthy · kept 0.6118 vs reference 0.7355
0.5M-no_attention-s4
48 epochs · 5,136 steps · healthy · kept 0.6108 vs reference 0.7355
0.5M-history_only-s0
20 epochs · 2,140 steps · not learning · kept 0.7547 vs reference 0.7355
- Wiring check on the real data (CPU)Before the pre-launch audit fixes; superseded by the checks below.passedjob
- Acceptance tests, incl. planted pair (must be found) and destroyed pairs (must show no gain)Run again inside every job before any training.passed
- Panel cache built and fingerprinted (CPU)205,417 panels, fingerprint 8519cf745f892b6f; 23 acceptance tests passed inside the job first.passedjob
- GPU preflight: measured step time and memoryAll 10 size/version checks ok on one H200; measured worst case under 1 hour for 30 slots.passedjob
- Training campaign: 30 slotsApproved 19 Oct. Watch it learn follows the run's own progress file.runningjob
03 · results
Four separate questions
Beating the tree, real pairs mattering, the shared-failure head, and new forecasters are judged separately. Each source gets its own verdict.
| Source | neural advantage | relational advantage | shared-failure advantage | transfer (new forecasters) |
|---|---|---|---|---|
| 0.5M · crypto | yes | yes | no | not enough evidence |
| 0.5M · economy-ecb | yes | no | no | no |
| 0.5M · economy-philfed | no | no | yes | not enough evidence |
| 0.5M · forecastbench | no | no | yes | no |
| 0.5M · numinous | no | no | not enough evidence | no |
| 0.5M · weather-deterministic | not enough evidence | not enough evidence | no | not enough evidence |
| 0.5M · weather-ensemble | not enough evidence | not enough evidence | yes | not enough evidence |
Diagnostic: these weeks were already looked at. Confirmation needs new weeks after the weights are frozen.
04 · baseline to beat
The re-checked tree, v0.77r
Every input time-valid and every model calibrated the same way.
| Source | Independent questions | Track record | Re-checked tree | Change |
|---|---|---|---|---|
| Crypto (replayed methods) | 24,171 | 0.438 | 0.403 | 8% lower error |
| Economy, ECB | 45 | 0.648 | 0.632 | too small to judge |
| Economy, Philadelphia Fed | 42 | 0.625 | 0.642 | too small to judge |
| ForecastBench | 1,056 | 0.396 | 0.397 | 0% lower error |
| Numinous + market price | 49 | 0.703 | 0.679 | too small to judge |
| Weather, single-run | 3,466 | 0.348 | 0.276 | 21% lower error |
| Weather, ensemble | 15,920 | 0.638 | 0.566 | 11% lower error |
Log loss, lower is better. The earlier ~25% ForecastBench gain was withdrawn after the leak fix.
Earlier runsv0.70 to v0.75, stopped+
v0.75 · stopped
00 · done
Fix v0.74's weak spots
Calibration only from answers known in time; separate slices for choosing and for ranges.
01 · done
Freeze one audited benchmark
Independent outcomes counted, copies kept together, unproven inputs refused.
02 · done
Does sharing help at all?
Small model on CPU, five matched versions. Must beat one-model-per-source on two main sources.
03 · next
Full run on an H200
Starts by itself only if stage 2 passes. Same rules, larger model.
04 · next
Prove it on new outcomes
Frozen policy scored on answers that arrive after the freeze. Then your approval.
Test finished: sharing one model across sources did not beat one model per source, so the full run did not start. Next: the Failure Atlas tests who misses together.
- AI crowd × Polymarketinsufficient evidencevs team —vs per-source —vs v0.74b —
- COVID-19team forecast keptvs team +0.0%vs per-source +0.0%vs v0.74b +0.0%
- Crypto pricesinsufficient evidencevs team —vs per-source —vs v0.74b —
- ECB economistsinsufficient evidencevs team +19.4%vs per-source −0.8%vs v0.74b +1.2%
- Ensemble weather, pressureinsufficient evidencevs team —vs per-source —vs v0.74b —
- Ensemble weather, raininsufficient evidencevs team —vs per-source —vs v0.74b —
- Ensemble weather, temperatureinsufficient evidencevs team −0.5%vs per-source −3.3%vs v0.74b −2.2%
- Ensemble weather, windinsufficient evidencevs team +3.3%vs per-source −1.1%vs v0.74b −1.0%
- Fluteam forecast keptvs team +0.0%vs per-source +0.0%vs v0.74b −0.4%
- ForecastBench questionsinsufficient evidencevs team +0.0%vs per-source +1.5%vs v0.74b +0.0%
- Philadelphia Fed economiststeam forecast keptvs team +1.7%vs per-source +1.7%vs v0.74b +0.5%
- Philadelphia Fed probabilitiesteam forecast keptvs team +16.7%vs per-source −6.6%vs v0.74b −6.6%
- Polymarket tradersinsufficient evidencevs team —vs per-source —vs v0.74b —
- Prediction marketsinsufficient evidencevs team —vs per-source —vs v0.74b —
- RSVinsufficient evidencevs team −2.3%vs per-source −2.6%vs v0.74b −4.4%
- Sea conditionsteam forecast keptvs team +1.7%vs per-source +0.0%vs v0.74b +0.9%
- Single-run weather, pressureinsufficient evidencevs team +0.0%vs per-source −9.4%vs v0.74b −7.6%
- Single-run weather, pressure (on-time answers only)insufficient evidencevs team +0.0%vs per-source −9.4%vs v0.74b −7.6%
- Single-run weather, raininsufficient evidencevs team +10.1%vs per-source −0.9%vs v0.74b +6.5%
- Single-run weather, rain (on-time answers only)insufficient evidencevs team +10.1%vs per-source −0.9%vs v0.74b +6.5%
- Single-run weather, temperatureteam forecast keptvs team −2.5%vs per-source −2.5%vs v0.74b −2.3%
- Single-run weather, temperature (on-time answers only)team forecast keptvs team −2.5%vs per-source −2.5%vs v0.74b −2.3%
- Single-run weather, windshared adoptedvs team +2.3%vs per-source −0.4%vs v0.74b +2.8%
- Single-run weather, wind (on-time answers only)shared adoptedvs team +2.3%vs per-source −0.4%vs v0.74b +2.8%
- Space weatherinsufficient evidencevs team —vs per-source —vs v0.74b —
Check 2 error of the shared model compared with each rival; positive = smaller error. Verdicts need reliable gains on both checks, grouped into two-day weather events.
v0.74 · completed, not promoted
- AI crowd × PolymarketNot enough evidencecheck 1 —check 2 —final —
- COVID-19Team forecast keptcheck 1 −9.3%check 2 −20.3%final −57.5%
- Crypto pricesTeam forecast keptcheck 1 +0.0%check 2 +0.0%final +0.0%
- ECB economistsNot enough evidencecheck 1 —check 2 —final —
- Ensemble weather, pressureNot enough evidencecheck 1 —check 2 —final —
- Ensemble weather, rainNot enough evidencecheck 1 —check 2 —final —
- Ensemble weather, temperatureModel ahead on checks 1–2check 1 +1.9%check 2 +5.0%final −1.1%
- Ensemble weather, windModel ahead on checks 1–2check 1 +8.3%check 2 +4.2%final +7.8%
- FluTeam forecast keptcheck 1 −2.2%check 2 −1.7%final −4.1%
- ForecastBench questionsNot enough evidencecheck 1 +0.9%check 2 +0.6%final −1.2%
- Philadelphia Fed economistsTeam forecast keptcheck 1 −6.8%check 2 +2.4%final −7.2%
- Philadelphia Fed probabilitiesTeam forecast keptcheck 1 +37.3%check 2 −34.7%final +2.4%
- Polymarket tradersTeam forecast keptcheck 1 +20.0%check 2 +0.8%final +0.1%
- Prediction marketsTeam forecast keptcheck 1 +0.3%check 2 +0.7%final +0.4%
- RSVNot enough evidencecheck 1 −5.3%check 2 +3.0%final +10.6%
- Sea conditionsTeam forecast keptcheck 1 +0.0%check 2 +0.0%final +0.0%
- Single-run weather, pressureNot enough evidencecheck 1 +3.2%check 2 +5.5%final +4.5%
- Single-run weather, pressure (on-time answers only)Not enough evidencecheck 1 +3.2%check 2 +5.5%final +4.5%
- Single-run weather, rainNot enough evidencecheck 1 +7.8%check 2 +7.8%final +1.3%
- Single-run weather, rain (on-time answers only)Not enough evidencecheck 1 +7.8%check 2 +7.8%final +1.3%
- Single-run weather, temperatureTeam forecast keptcheck 1 −1.8%check 2 −3.2%final −4.3%
- Single-run weather, temperature (on-time answers only)Team forecast keptcheck 1 −1.7%check 2 −1.1%final −0.9%
- Single-run weather, windTeam forecast keptcheck 1 −1.5%check 2 −1.4%final −2.1%
- Single-run weather, wind (on-time answers only)Model ahead on checks 1–2check 1 +4.7%check 2 +3.2%final +4.3%
- Space weatherTeam forecast keptcheck 1 +4.5%check 2 +17.1%final +17.2%
Gain = how much smaller the model's error is than the team's own forecast. Check 1 and 2 are two later periods; the model is only kept where both are reliably ahead of the team and of a strong tree benchmark. Final = v0.74's last weeks, scored only after every verdict was locked; they have now been opened, so v0.75 treats them as history, not as a blind test.
Reading the active run…
01 · the test
What would count as learning?
Forecast gain, range calibration and fragility warnings need separate evidence.
Dependence helps
Full memory must beat no history, no dependence and shuffled dependence on every seed.
Beats simple methods
Gain over the strongest simple comparison, with a 95% range above zero.
Independent evidence
Only periods with earlier resolved outcomes can earn a verdict; reference-only periods stay explicit.
- v0.57 stays served
- nothing publishes on its own
- head-to-head vs v0.57 before any switch
02 · versions
This run’s versions and seeds
Four matched versions; the current attempt’s seeds and completion come from its own records.
- 01 · part 1
No history
panel encoder
- 02 · part 1
Track records
+ causal records, pair co-miss, lineage families
- 03 · part 1
Shuffled records
control for the records
- 04 · part 2
Full memory
+ sequence memory (GRU)
- 05 · part 3
Full + pretraining
+ self-supervised pretraining
03 · learning curves
How each version learns
Saved training loss and sparklines from this attempt only, one line per seed. Curves arrive when a slot publishes; the page checks every minute.
Reading the run…
04 · checks
Before it learns
The current attempt’s fingerprint, source eligibility and learning receipts.
- Frozen snapshot
- awaiting receipt
- Learning proof
- awaiting receipt
- Saved weights reproduce
- awaiting receipt
snapshot unavailable
New weather covers one five-day stretch, not five independent test periods. Raw rows are not independent questions.
Pre-training eligibility is not proof of improvement or completed optimizer updates. Polymarket trades, short crypto overlap and satellite sunlight remain excluded from training.
04 · corpus
What it learns from, by domain
Every source and its role in this run.
Reading the run…
05 · per source
Source by source
How many seeds pass for each version.
Reading the run…
06 · this version
The shared-failure architecture
Shared encoder, source-specific forecast heads and a dedicated failure branch; about 0.5M weights in the full version.
01 · panel
Forecasts in context
Up to 64 members; task, horizon, scale, lineage and decision time.
02 · memory
Who fails together?
Time-valid individual records, shared-failure links and sequence memory.
03 · shared representation
A model of forecasters
Shared encoder with source-specific forecast heads and a dedicated failure branch.
04 · separate outputs
Three independent verdicts
Forecast correction, calibrated ranges and confident-failure warnings.
True-answer scoring
Model and comparison use identical outcomes; unstable yes/no loss is repaired.
Ranges without a widening cap
Fitted on practice data only; coverage and sharpness still need separate evidence.
Learning checked before training
Forecast and failure gradients, history/pair influence and save/reload parity are checked on real cases.
Full memory
panel + causal history + shared-failure links + sequence memory
No history
today’s panel without past outcomes
No shared failures
keeps individual skill; removes shared-failure links
Shuffled shared failures
keeps individual records; relabels co-failure partners
07 · untouched window
Four separate capability tests
The full run reserves an untouched question window before any training or model selection.
Weather final decisions sealed from — · rehearsals are not release evidence.
Forecast gain
Matched outcomes against the strongest simple method, v0.68 and served v0.57; comparisons not yet completed.
Useful ranges
Coverage and sharpness, scored on the same cases as the reference.
Fragility warnings
Failures caught in a fixed 10% review queue; false alarms reported separately.
Shared-failure learning
Full memory against controls removing or shuffling dependence; no claim before matched judging.
Older notesv0.60 to v0.66, AI crowd+
08 · previous runs
Earlier runs, plainly
Why nothing after v0.57 was released.
- Crypto final check (12 Oct)Failed. Only 4 independent weeks in the sealed final period (10 required); −4.7% [−17.7%, +3.7%] after widening; ranges covered 97.1% (too wide); 0 of 108 confident failures warned. Nothing promoted; v0.57 stays served. The warning comparison had a defect since fixed, so its number is diagnostic only.
- v0.66Stopped on 10 Oct after 11 of 15 trainings. Most sources did worse than the simple comparison (seed 11: COVID-19 −16%, SPF probabilities −86%); only crypto (+0.5 to +2.7%) and space weather (+1 to +5%) were ahead without history. Real track records did worse than shuffled ones. Nothing was released.
- v0.65Crypto passed on all three seeds (32,322 outcomes, 105 weeks, about 1.2–1.4% better). Track records did not beat shuffled ones; nothing was released.
- v0.64Seven of 19 trainings finished before one crashed. The audit found leakage between sources and no source passing the strict test; nothing was released.
- v0.63Crypto looked about 8.7% better, but ranges covered about 88% instead of 90% and one seed failed; not released.
08 · numinous
Learning from an AI crowd
323 AI forecasters: who agrees, who's redundant, who fails together.
Track records
339,667
dated forecasts from 323 AI forecasters on 375 markets
Shared blind spots
3,707
forecaster pairs wrong together at least once · 206 pairs 3+ times · lower bound (top-40 panel)
Confidently wrong
40
of 3,280 final forecasts on 82 resolved markets: the rare failures Zeno learns to see coming
The test
With vs without Numinous, on identical questions: measuring. The without-Numinous baseline (v0.56.1) is finished; no gain is shown until the matched run scores.
Next together
Full panel beyond the top 40 (adds blind-spot pairs, never removes them) · on-demand forecasts once API access is set up.
Previous runs
v0.62 canceled, v0.61 failed closed
v0.62 was stopped mid-run: its release test could approve invalid claims and its data handling was not invariant, so its numbers are not evidence. v0.61 recorded no comparison scores, so none of its 26 claims could pass.
- Crypto0.927 → 0.862 · beats reference both periods · 3/3 seeds
- Ensemble weather0.920 → 0.889 · beats reference both periods · 3/3 seeds
- COVID-190.874 → 0.805 · beats reference both periods · 3/3 seeds
- RSV1.065 → 1.061 · mixed across seeds · 1/3 seeds
- Flu0.623 → 0.623 · reference kept · 0/3 seeds
- Markets0.338 → 0.338 · reference kept · 0/3 seeds
- US survey (SPF)0.809 → 0.824 · reference kept · 0/3 seeds
- Sea0.721 → 0.702 · beats reference both periods · 3/3 seeds
- Deterministic weather1.107 → 1.041 · beats reference both periods · 3/3 seeds
Currently servingv0.57, the public model, not this run+
v0.57 · rounds
v0.57 training, round by round
Teal beats the reference, orange loses to it.
Waiting for the first training round.
gate
What it must beat
The fixed ladder every release faces.
reading the baseline comparison…
What would prove it wrong · 1 of 10 measured
- With vs without track recordsmeasured · histories currently hurt
- With vs without Numinousrunning · baseline finished; matched run pending
- Reference vs GBM vs Zeno, every sourcepartial · GBM checked on physical sources only
- Leave one domain outnot yet run · no fitting on the held-out domain
- No relational attentionnot yet run · does interaction add anything?
- No co-miss featuresnot yet run · tests the shared-blind-spot idea
- Shuffle forecaster identitynot yet run · forecasts kept, identities destroyed
- Shuffle historiesnot yet run · histories kept, owners destroyed
- Shuffle who failed togethernot yet run · skill kept, pairings destroyed
- Cloned crowd · regime shiftnot yet run · does Zeno discount copies and adapt?
Awaiting the first training round…
Historical technical record
Earlier research only · not v0.72 telemetry or release evidence.
v0561-nonum-a100-20261006T2037Z · finished 2026-10-07 · confirmation (never used to pick anything) · lower is better
Histories lost on all 3 seeds (they overfit). With: 0.809–0.815 · without: 0.797–0.801.
- · Markets reference repaired after the run (it ignored yes/no direction); the markets row uses the repaired reference, rescored from saved predictions.
- · Per-source acceptance was fitted on the development period only; the confirmation period is a report, not a second gate.
- · Seed spread is not a confidence interval.
- · Development-period headline scores use the better of model and reference per source and are optimistic; this table uses confirmation scores.
Sources appear once the run has read its data…
Comparator scores appear when the first arm finishes scoring…
Each seed appears here as it finishes…
01
Track records
Each forecaster's earlier errors and coverage, in time order.
—
02
Missed together
Which forecasters failed on the same outcomes before.
—
03
Behaviour pretraining
Learns next error and next coverage from history, never the outcome.
—
04
Task context
Asset, location, target, horizon, units and season, joined into one hashed label.
—
05
Correction + warning
A corrected outcome forecast on top of the reference; from v0.50 the warning is read off that forecast, not a separate part.
—
corpus census appears once the run publishes it
Arena · modelv057 · gate-passed per source
Download & model details ↗Evidence & limits
- In service
- Served model
- Model
- v057 · v0451 schema 1
- Gate-passed sources
- 5 of 10
- Warnings
- ensemble weather only
Gate-passed per source on dev and the later confirmation period; every other source returns its reference, labelled.
Derived warnings beat the old warning part on four of five sources, and beat the baseline on RSV and ensemble weather in both unseen periods. Flu is a small step back.
Family members share blind spots: matched on skill, range width and difficulty, the gap stays clear in earlier weeks (COVID +10, flu +18, RSV +24 points) but shrinks to a few points and becomes uncertain in the latest weeks. Corrected after an audit found one input read the outcome: as a warning add-on it does not pass (only RSV improves, and near chance), so it stays a check and the plain average stays live.
reading the shadow experiment…