John Faig

🚩 When Polymarket or Kalshi says there's a 70% chance something will happen, should we believe it? Excited to share our new paper, just published in PNAS, in which we examined more than ONE MILLION forecasts from Polymarket, Kalshi, Manifold, and Metaculus, spanning 12 domains and forecasting horizons ranging from hours to years. The main finding is striking: Prediction markets and collective forecasting platforms are remarkably well calibrated. When they assign events a 70% probability, those events really do happen about 70% of the time. And this pattern holds surprisingly well across different platforms, domains, and forecasting horizons. But there's an important twist. When we looked more closely, we discovered systematic biases in some of the domains where forecasting matters most (wars, elections). For decades, psychologists have documented how poorly humans often reason about probability and uncertainty. Our findings offer an encouraging counterpoint: When judgments and information are aggregated, we can achieve a surprisingly accurate understanding of an uncertain future. That doesn't mean we can predict what will happen next with certainty. But it does mean that we can be remarkably good at estimating how uncertain the future really is. This work was a great collaboration. The data for this project was meticulously collected for years by Justin Dickerson, the project was led by Hilla Shinitzky and Maayan Peer, with @mattansb leading the analyses. 📄 Read the paper: pnas.org/doi/10.1073/pnas.26… #Forecasting #Polymarket #Kalshi #DecisionScience #PNAS