Psychological Strategies for Winning a Geopolitical Forecasting Tournament

Mellers, Barbara A.; Ungar, Lyle H.; Baron, Jonathan; Ramos, Jaime; Gürçay, Burcu; Fincher, Katrina; Scott, Sydney E.; Moore, Don A.; Atanasov, Pavel; Swift, Samuel A.; Murray, Terry; Stone, Eric; Tetlock, Philip E.

doi:10.1177/0956797614524255

Cited by 253 publications

(264 citation statements)

References 39 publications

Supporting

Mentioning

225

Contrasting

Order By: Relevance

“…Figure 4 presents calibration curves for each of our four conditions, accompanied by the variability between individual forecasters around the average. Additional comparisons of resolution scores across our experimental conditions are presented by Mellers et al (2014).…”

Section: Resultsmentioning

confidence: 99%

“…We also omit data from the select group of "superforecasters" in years 2 and 3. For more information about these other conditions, see Mellers et al (2014). For more detail about the prediction-market conditions, see Atanasov et al (2017).…”

Section: Endnotesmentioning

confidence: 99%

“…Three other papers examine the conditions that contribute to individual forecasting accuracy. Mellers et al (2014) provide an overview of the forecasting tournament and discuss the positive impacts of three behavioral interventions-training, teaming, and trackingon individual performance in prediction polls using data from the first two years of the tournament. Mellers et al (2014) do not, however, examine confidence and accuracy over time to understand how they develop.…”

Section: The Present Researchmentioning

confidence: 99%

“…Mellers et al (2014) provide an overview of the forecasting tournament and discuss the positive impacts of three behavioral interventions-training, teaming, and trackingon individual performance in prediction polls using data from the first two years of the tournament. Mellers et al (2014) do not, however, examine confidence and accuracy over time to understand how they develop. Mellers et al (2015a) explore the profiles of individual forecasters using dispositional, situational, and behavioral variables.…”

Section: The Present Researchmentioning

confidence: 99%

See 3 more Smart Citations

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Swift

Minster²,

Mellers³

et al. 2016

Preprint

View full text Add to dashboard Cite

This research examines the development of confidence and accuracy over time in the context of forecasting. Although overconfidence has been studied in many contexts, little research examines its progression over long periods of time or in consequential policy domains. This study employs a unique data set from a geopolitical forecasting tournament spanning three years in which thousands of forecasters predicted the outcomes of hundreds of events. We sought to apply insights from research to structure the questions, interactions, and elicitations to improve forecasts. Indeed, forecasters’ confidence roughly matched their accuracy. As information came in, accuracy increased. Confidence increased at approximately the same rate as accuracy, and good calibration persisted. Nevertheless, there was evidence of a small amount of overconfidence (3%), especially on the most confident forecasts. Training helped reduce overconfidence and team collaboration improved forecast accuracy. Together, teams and training reduced overconfidence to 1%. Our results provide reason for tempered optimism regarding confidence calibration and its development over time in consequential field contexts.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Endnotesmentioning

confidence: 99%

Section: The Present Researchmentioning

confidence: 99%

Section: The Present Researchmentioning

confidence: 99%

See 2 more Smart Citations

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Swift

Minster²,

Mellers³

et al. 2016

Preprint

View full text Add to dashboard Cite

show abstract

“…However, despite its potential to boost accuracy in a wide range of contexts, including lie detection, political forecasting, investment decisions, and medical decision making (8)(9)(10)(11)(12)(13)(14), little is known about the conditions that underlie the emergence of collective intelligence in real-world domains. Which features of decision makers and decision contexts favor the emergence of collective intelligence?…”

mentioning

confidence: 99%

Boosting medical diagnostics by pooling independent judgments

Kurvers

Herzog

Hertwig

et al. 2016

Proc. Natl. Acad. Sci. U.S.A.

152

140

View full text Add to dashboard Cite

Collective intelligence refers to the ability of groups to outperform individual decision makers when solving complex cognitive problems. Despite its potential to revolutionize decision making in a wide range of domains, including medical, economic, and political decision making, at present, little is known about the conditions underlying collective intelligence in real-world contexts. We here focus on two key areas of medical diagnostics, breast and skin cancer detection. Using a simulation study that draws on large real-world datasets, involving more than 140 doctors making more than 20,000 diagnoses, we investigate when combining the independent judgments of multiple doctors outperforms the best doctor in a group. We find that similarity in diagnostic accuracy is a key condition for collective intelligence: Aggregating the independent judgments of doctors outperforms the best doctor in a group whenever the diagnostic accuracy of doctors is relatively similar, but not when doctors' diagnostic accuracy differs too much. This intriguingly simple result is highly robust and holds across different group sizes, performance levels of the best doctor, and collective intelligence rules. The enabling role of similarity, in turn, is explained by its systematic effects on the number of correct and incorrect decisions of the best doctor that are overruled by the collective. By identifying a key factor underlying collective intelligence in two important real-world contexts, our findings pave the way for innovative and more effective approaches to complex real-world decision making, and to the scientific analyses of those approaches.collective intelligence | groups | medical diagnostics | dermatology | mammography C ollective intelligence, that is, the ability of groups to outperform individual decision makers when solving complex cognitive problems, is a powerful approach for boosting decision accuracy (1-7). However, despite its potential to boost accuracy in a wide range of contexts, including lie detection, political forecasting, investment decisions, and medical decision making (8-14), little is known about the conditions that underlie the emergence of collective intelligence in real-world domains. Which features of decision makers and decision contexts favor the emergence of collective intelligence? Which decision-making rules permit this potential to be harnessed? We here provide answers to these important questions in the domain of medical diagnostics.Our work builds on recent findings on combining decisions, a research paradigm known as "two heads better than one" (15)(16)(17)(18)(19)(20). In their seminal study, Bahrami et al. (15) showed that two individuals permitted to communicate freely while engaging in a visual perception task, achieved better results than the better of the two did alone. Koriat (17) subsequently demonstrated that this collective intelligence effect also emerges in the absence of communication when the "maximum-confidence slating algorithm" (hereafter called confidence rule) is used and th...

show abstract

Decision Making in Groups and Organizations

Tindale

Kluwe

2015

The Wiley Blackwell Handbook of Judgment and Decision Making

View full text Add to dashboard Cite

Psychological Strategies for Winning a Geopolitical Forecasting Tournament

Cited by 253 publications

References 39 publications

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

Boosting medical diagnostics by pooling independent judgments

Decision Making in Groups and Organizations

Contact Info

Product

Resources

About