Anastasi Lecture 2016

Improving Human Forecasts of Geopolitical Events

Barbara Mellers, University of Pennsylvania

I report the highlights of four recent forecasting tournaments designed to uncover the most accurate ways of eliciting and combining forecasts from a large, dispersed crowd. Over 30,000 participants from around the world estimated the likelihood of geopolitical events ranging from pandemics and military conflicts to international agreements and refugee flows. Participants were randomly assigned to experimental conditions that tested elicitation methods (surveys vs prediction markets), social dynamics (forecasters working alone vs in teams), instruction in probabilistic reasoning (training vs no training) and a variety of algorithms for aggregating forecasts. At the end of each year, the top 2% of forecasters were assigned to elite teams of “superforecasters”. Superforecasters beat intelligence analysts at their own game by making forecasts that were 30% more accurate on the same questions with classified information. In separate laboratory tests, superforecasters were better than comparison groups on additional benchmarks of good judgment. To improve forecasts, we recommend combinations of behavioral and statistical interventions that identify and cultivate the very best in human judgment.

Barbara Mellers is the George I. Heyman University Professor, University of Pennsylvania with cross appointments in the Marketing Department at the Wharton School and the Psychology Department in the School of Arts and Sciences.
Mellers was trained in quantitative psychology. For many years, she studied how people make judgements and decisions that violate principles of rationality. She examined what we perceive as fair and why, how emotions and surprise influence choice, and how the context and the question format shape our judgements and decisions. More recently, her focus has been on how to make human judgements better. She is currently working on a new tournament called The Foresight Project and two projects with the intelligence community that explore ways to improve reasoning and combine human forecasts with machine data.