For teachers: how to read AI forecasts and check them

Glossary

Brier

A score for the accuracy of a probability forecast. 0 is perfect, 0.25 is the same as always saying 50%, 1 is sure and wrong. Lower is better. ForecastBench yardsticks: superforecasters 0.086, best AI models 0.103, a coin (always 50%) 0.25.

ECE, calibration

Whether confidence matches how often the answer is right: if an AI says "70%" about a hundred cases, are about seventy right. Best ECE today: 0.022 (nova).

Effective voices

How many truly independent opinions a group holds. The AIs share training data, so 14 models count as about 2.0.

"Council" and "crowd"

The Council is the joint decision of the AIs (the majority side). The crowd is the price of a close question at Polymarket or Kalshi, where people bet real money.

Sealed before the open

The forecast is written down and gets a hash and an external timestamp before the market opens, so it cannot be changed after the fact.

Virtual money

An account of an imaginary $100,000: no real money. Since October 19: holding cash $100,258 · a 60/40 stock/bond portfolio $99,594 · the S&P 500 (bought and held) $98,885 · the Council $97,069.

45-minute lesson plan

1. 0-5 min. Hypothesis
   The class writes a hypothesis: "do AIs call markets better than a coin?" and what counts as "better".
2. 5-15 min. Dispute of the day
   Open the live page and find the "Dispute of the day": today it is "fear index (VIX)". Which AIs say up, which say down?
3. 15-25 min. Your own call
   Before seeing the AIs, each student picks ▲ up or ▼ down and a confidence of 55-95% (this is how the "Your call before the AIs" block on the page works) and writes it on paper with the time.
4. 25-35 min. Count the Brier
   For one forecast: (probability - outcome)². Outcome 1 is up, 0 is down. Compare with the coin 0.25 and the table on the page. The page sample: 91 questions.
5. 35-45 min. Discussion
   In the evening, after the close (4:00 PM ET), check the notes against the result of the day. What does "the AI got it right" mean? Is one day enough? Why seal before the open, and how many independent voices do 14 models have?

Tasks: (1) compute your own Brier for a week from your notes; (2) find the best-calibrated AI in the page table and explain it in your own words; (3) devise a way to tell skill from luck on 91 questions. Tasks are for learning: no money and no promise of returns.