Improving Judgments of Existential Risk: Better Forecasts, Questions, Explanations, Policies

Karger, Ezra; Atanasov, Pavel

doi:10.2139/ssrn.4001628

Cited by 7 publications

(17 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our experiment, we also elicited meta-predictions (Martinie et al, 2020;Wilkening et al, 2022). Just like we did with the forecasts, we also properly incentivized the meta-predictions, thus we can directly score them in real time, using actual aggregate crowd judgments, much in the same way reciprocal scoring methods work (Karger et al, 2021(Karger et al, , 2022.…”

Section: Meta-predictions and Accuracy Incentivesmentioning

confidence: 99%

“…This required each forecaster to make two predictions about each question: one about their own beliefs and one about the beliefs of others. Other researchers have used the judgments of forecasters known to be highly accurate as intersubjective criteria (Karger et al, 2021(Karger et al, , 2022. There may be cases in which judges are all similarly compensated for performing a variety of tasks or are incentivized for ground truth performance regardless of how they are evaluated for nonincentivized purposes.…”

Section: Forecasts Versus Meta-predictionsmentioning

confidence: 99%

“…A separate line of research uses a slightly different intersubjective judgment, referred to as reciprocal scoring, as a method for scoring the accuracy of forecasts when event resolutions may occur at extremely distant time horizons, or be unknowable entirely (Karger et al, 2021(Karger et al, , 2022. In these studies, forecasters are typically asked what they think highly qualified and extremely accurate forecasters might predict about such unresolvable events.…”

Section: Meta-predictions and Accuracy Incentivesmentioning

confidence: 99%

“…This creates a real challenge in time-sensitive settings, where obtaining accurate predictions cannot wait until decision makers calibrate their expectations about the accuracy of their forecasters. This might include settings where forecasting problems have prohibitively long time horizons, such as long-term economic trends (Himmelstein, Budescu & Han, 2022); forecasts about life and death issues such as global health or climate crises (Ho et al, 2015(Ho et al, , 2019Taylor & Taylor, 2023); prognosis and prediction of disease treatment outcomes (Ioannidis, 2009); assessing the performance of professional athletes to optimize roster construction (Lee et al, 2018;Lewis, 2004;Miller & Sanjurjo, 2018); identification of the importance and replicability of scientific research (Aczel et al, 2021;Camerer et al, 2016); or even events that may never have a clear resolution (Karger et al, 2021(Karger et al, , 2022. Analyzing traits known to be related to forecasting skills (Colson & Cooke, 2018;Ho, 2020;Mellers, Stone, Atanasov, et al, 2015) or behaviors that occur during forecasting elicitation (Atanasov et al, 2020) can provide a head start, but these methods ultimately pale in comparison to having information about a forecaster's past accuracy (Himmelstein et al, 2021).…”

mentioning

confidence: 99%

See 3 more Smart Citations

The wisdom of many in few: Finding individuals who are as wise as the crowd.

Himmelstein¹,

Budescu²,

Ho³

2023

Journal of Experimental Psychology: General

View full text Add to dashboard Cite

Is forecasting ability a stable trait? While domain knowledge and reasoning abilities are necessary for making accurate forecasts, research shows that knowing how accurate forecasters have been in the past is the best predictor of future accuracy. However, unlike the measurement of other traits, evaluating forecasting skill requires substantial time investment. Forecasters must make predictions about events that may not resolve for many days, weeks, months, or even years into the future before their accuracy can be estimated. Our work builds upon methods such as cultural consensus theory and proxy scoring rules to show talented forecasters can be discriminated in real time, without requiring any event resolutions. We define a peer similarity-based intersubjective evaluation method and test its utility in a unique longitudinal forecasting experiment. Because forecasters predicted all events at the same points in time, many of the confounds common to forecasting tournaments or observational data were eliminated. This allowed us to demonstrate the effectiveness of our method in real time, as time progressed and more information about forecasters became available. Intersubjective accuracy scores, which can be obtained immediately after the forecasts are made, were both valid and reliable estimators of forecasting talent. We also found that asking forecasters to make meta-predictions about what they expect others to believe can serve as an incentive-compatible method of intersubjective evaluation. Our results indicate that selecting small groups of, or even single forecasters, based on intersubjective accuracy can yield subsequent forecasts that approximate the actual accuracy of much larger crowd aggregates.

show abstract

Section: Meta-predictions and Accuracy Incentivesmentioning

confidence: 99%

Section: Forecasts Versus Meta-predictionsmentioning

confidence: 99%

Section: Meta-predictions and Accuracy Incentivesmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

The wisdom of many in few: Finding individuals who are as wise as the crowd.

Himmelstein¹,

Budescu²,

Ho³

2023

Journal of Experimental Psychology: General

View full text Add to dashboard Cite

show abstract

“…Promising applications of intersubjective measures include skill identification and incentive provision (Karger, Atanasov & Tetlock, 2022). At the same time, as Himmelstein, Budescu, and Ho (2022) point out, intersubjective measures that relate an individual's estimates to the consensus may be limited in their utility in spotting accurate forecasters with unique views.…”

Section: Research Synthesismentioning

confidence: 99%

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein²

2022

Preprint

View full text Add to dashboard Cite

Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: 1) accuracy-related measures, such as proper scores, model-based estimates of accuracy and excess volatility scores; 2) intersubjective measures, including proxy, surrogate and similarity scores; 3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; 4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and 5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term, existential risk prediction and corporate forecasting tournaments.

show abstract

Long‐range subjective‐probability forecasts of slow‐motion variables in world politics: Exploring limits on expert judgment

TETLOCK

Karvetski²,

Satopää

et al. 2023

Futures & Foresight Science

View full text Add to dashboard Cite

Skeptics see long‐range geopolitical forecasting as quixotic. A more nuanced view is that although predictability tends to decline over time, its rate of descent is variable. The current study gives geopolitical forecasters a sporting chance by focusing on slow‐motion variables with low base rates of change. Analyses of 5, 10, and 25‐year cumulative‐risk judgments made in 1988 and 1997 revealed: (a) specialists beat generalists at predicting nuclear proliferation but not shifting nation‐state boundaries; (b) some counterfactual interventions—for example, Iran gets the bomb before 2022—boosted experts’ edge but others—for example, nuclear war before 2022—eliminated it; (c) accuracy fell faster on topics where expertise conferred no edge in shorter‐range forecasts. To accelerate scientific progress, we propose adversarial collaborations in which clashing schools of thought negotiate Bayesian reputational bets on divisive issues and use Lakatosian scorecards to incentivize the honoring of bets.

show abstract

Improving Judgments of Existential Risk: Better Forecasts, Questions, Explanations, Policies

Cited by 7 publications

References 44 publications

The wisdom of many in few: Finding individuals who are as wise as the crowd.

The wisdom of many in few: Finding individuals who are as wise as the crowd.

Talent Spotting in Crowd Prediction

Long‐range subjective‐probability forecasts of slow‐motion variables in world politics: Exploring limits on expert judgment

Contact Info

Product

Resources

About