Reciprocal Scoring: A Method for Forecasting Unanswerable Questions

Karger, Ezra; Monrad, Joshua Teperowski; Mellers, Barb

doi:10.2139/ssrn.3954498

Cited by 6 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In our experiment, we also elicited meta-predictions (Martinie et al, 2020;Wilkening et al, 2022). Just like we did with the forecasts, we also properly incentivized the meta-predictions, thus we can directly score them in real time, using actual aggregate crowd judgments, much in the same way reciprocal scoring methods work (Karger et al, 2021(Karger et al, , 2022.…”

Section: Meta-predictions and Accuracy Incentivesmentioning

confidence: 99%

“…This required each forecaster to make two predictions about each question: one about their own beliefs and one about the beliefs of others. Other researchers have used the judgments of forecasters known to be highly accurate as intersubjective criteria (Karger et al, 2021(Karger et al, , 2022. There may be cases in which judges are all similarly compensated for performing a variety of tasks or are incentivized for ground truth performance regardless of how they are evaluated for nonincentivized purposes.…”

Section: Forecasts Versus Meta-predictionsmentioning

confidence: 99%

“…A separate line of research uses a slightly different intersubjective judgment, referred to as reciprocal scoring, as a method for scoring the accuracy of forecasts when event resolutions may occur at extremely distant time horizons, or be unknowable entirely (Karger et al, 2021(Karger et al, , 2022. In these studies, forecasters are typically asked what they think highly qualified and extremely accurate forecasters might predict about such unresolvable events.…”

Section: Meta-predictions and Accuracy Incentivesmentioning

confidence: 99%

“…This creates a real challenge in time-sensitive settings, where obtaining accurate predictions cannot wait until decision makers calibrate their expectations about the accuracy of their forecasters. This might include settings where forecasting problems have prohibitively long time horizons, such as long-term economic trends (Himmelstein, Budescu & Han, 2022); forecasts about life and death issues such as global health or climate crises (Ho et al, 2015(Ho et al, , 2019Taylor & Taylor, 2023); prognosis and prediction of disease treatment outcomes (Ioannidis, 2009); assessing the performance of professional athletes to optimize roster construction (Lee et al, 2018;Lewis, 2004;Miller & Sanjurjo, 2018); identification of the importance and replicability of scientific research (Aczel et al, 2021;Camerer et al, 2016); or even events that may never have a clear resolution (Karger et al, 2021(Karger et al, , 2022. Analyzing traits known to be related to forecasting skills (Colson & Cooke, 2018;Ho, 2020;Mellers, Stone, Atanasov, et al, 2015) or behaviors that occur during forecasting elicitation (Atanasov et al, 2020) can provide a head start, but these methods ultimately pale in comparison to having information about a forecaster's past accuracy (Himmelstein et al, 2021).…”

mentioning

confidence: 99%

See 3 more Smart Citations

The wisdom of many in few: Finding individuals who are as wise as the crowd.

Himmelstein¹,

Budescu²,

Ho³

2023

Journal of Experimental Psychology: General

View full text Add to dashboard Cite

Is forecasting ability a stable trait? While domain knowledge and reasoning abilities are necessary for making accurate forecasts, research shows that knowing how accurate forecasters have been in the past is the best predictor of future accuracy. However, unlike the measurement of other traits, evaluating forecasting skill requires substantial time investment. Forecasters must make predictions about events that may not resolve for many days, weeks, months, or even years into the future before their accuracy can be estimated. Our work builds upon methods such as cultural consensus theory and proxy scoring rules to show talented forecasters can be discriminated in real time, without requiring any event resolutions. We define a peer similarity-based intersubjective evaluation method and test its utility in a unique longitudinal forecasting experiment. Because forecasters predicted all events at the same points in time, many of the confounds common to forecasting tournaments or observational data were eliminated. This allowed us to demonstrate the effectiveness of our method in real time, as time progressed and more information about forecasters became available. Intersubjective accuracy scores, which can be obtained immediately after the forecasts are made, were both valid and reliable estimators of forecasting talent. We also found that asking forecasters to make meta-predictions about what they expect others to believe can serve as an incentive-compatible method of intersubjective evaluation. Our results indicate that selecting small groups of, or even single forecasters, based on intersubjective accuracy can yield subsequent forecasts that approximate the actual accuracy of much larger crowd aggregates.

show abstract

Section: Meta-predictions and Accuracy Incentivesmentioning

confidence: 99%

Section: Forecasts Versus Meta-predictionsmentioning

confidence: 99%

Section: Meta-predictions and Accuracy Incentivesmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

The wisdom of many in few: Finding individuals who are as wise as the crowd.

Himmelstein¹,

Budescu²,

Ho³

2023

Journal of Experimental Psychology: General

View full text Add to dashboard Cite

show abstract

“…This creates a real challenge in time sensitive settings, where obtaining accurate predictions cannot wait until decision makers calibrate their expectations about the accuracy of their forecasters. This might include settings where forecasting problems have prohibitively long time horizons, such as long term economic trends (Himmelstein, Budescu & Han, 2022); forecasts about life and death issues such as global health or climate crises (Ho et al, 2015(Ho et al, , 2019Taylor & Taylor, 2021); prognosis and prediction of disease treatment outcomes (Ioannidis, 2009); assessing the performance of professional athletes to optimize roster construction (Lee et al, 2018;Lewis, 2004;Miller & Sanjurjo, 2018); identification of the importance and replicability of scientific research (Aczel et al, 2021;Camerer et al, 2016); or even events that may never have a clear resolution (Karger et al, 2021(Karger et al, , 2022. Analyzing traits known to be related to forecasting skill (Colson & Cooke, 2018;Ho, 2020;Mellers, Stone, Atanasov, et al, 2015) or behaviors that occur during forecasting elicitation (Atanasov et al, 2020) can provide a head start, but these methods ultimately pale in comparison to having information about a forecaster's past accuracy (Himmelstein et al, 2021).…”

Section: Introductionmentioning

confidence: 99%

The wisdom of many in few: Finding individuals who are as wise as the crowd

Himmelstein,

Budescu,

2023

Preprint

View full text Add to dashboard Cite

Is forecasting ability a stable trait? While domain knowledge and reasoning abilities are necessary for making accurate forecasts, research shows that knowing how accurate forecasters have been in the past is the best predictor of future accuracy. However, unlike the measurement of other traits, evaluating forecasting skill requires substantial time investment. Forecasters must make predictions about events that may not resolve for many days, weeks, months, or even years into the future before their accuracy can be estimated. Our work builds upon methods such as cultural consensus theory and proxy scoring rules to show talented forecasters can be discriminated in real-time, without requiring any event resolutions. We define a peer similarity-based intersubjective evaluation method and test its utility in a unique longitudinal forecasting experiment. Because forecasters predicted all events at the same points in time, many of the confounds common to forecasting tournaments or observational data were eliminated. This allowed us to demonstrate the effectiveness of our method in real-time, as time progressed and more information about forecasters became available. Intersubjective accuracy scores, which can be obtained immediately after the forecasts are made, were both valid and reliable estimators of forecasting talent. We also found that asking forecasters to make meta-predictions about what they expect others to believe can serve as an incentive compatible method of intersubjective evaluation. Our results indicate that selecting small groups of, or even single forecasters, based on intersubjective accuracy can yield subsequent forecasts that approximate the actual accuracy of much larger crowd aggregates.

show abstract

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein

2023

International Series in Operations Research &Amp; Management Science

View full text Add to dashboard Cite

Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: 1) accuracy-related measures, such as proper scores, modelbased estimates of accuracy and excess volatility scores; 2) intersubjective measures, including proxy, surrogate and similarity scores; 3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; 4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and 5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term prediction and corporate forecasting tournaments.

show abstract

Reciprocal Scoring: A Method for Forecasting Unanswerable Questions

Cited by 6 publications

References 0 publications

The wisdom of many in few: Finding individuals who are as wise as the crowd.

The wisdom of many in few: Finding individuals who are as wise as the crowd.

The wisdom of many in few: Finding individuals who are as wise as the crowd

Talent Spotting in Crowd Prediction

Contact Info

Product

Resources

About