Measuring Forecasting Skill from Text

Zong, Shi; Ritter, Alan; Hovy, Eduard

doi:10.48550/arxiv.2006.07425

Cited by 2 publications

(3 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our Study 2 analysis focused on independent forecasters who were not incentivized to write detailed rationales, so we did not analyze linguistic features of rationales. However, strong results across multiple previous studies (Horowitz et al, 2019, Zong et al, 2020 show that that such can be very helpful in spotting consistently accurate forecasters in settings where inter-forecaster communication is encouraged.…”

Section: Research Synthesismentioning

confidence: 98%

“…The simplest analyses focus on the rationale length, measured by the number of words or number of characters. More sophisticated natural-language processing techniques (NLP) include bag-of-words and topic modeling, which analyze which words and phrases tend to co-occur together (Horowitz et al, 2019;Zong et al, 2020). 3 NLP techniques have also been used to measure latent psychological factors, such as forecasters' tendency to consider base rates, to focus on the future versus the past, and to engage in complex thought patterns .…”

Section: Behavioralmentioning

confidence: 99%

“…In a separate analysis based on data from Good Judgment Open, Zong et al ( 2020) also found that statements about uncertainty, and ones focused on the past relate to better accuracy, while statements focused on the future correspond with worse accuracy. Zong et al (2020) Cognitive Styles: In contrast to fluid intelligence measures, which are objective (performancebased), thinking style measures rely on forecasters to self-reflect and report on their own proclivities. Among these measures, actively open-minded thinking was the only one that has been significantly linked with better accuracy: reported a small but significant correlation (r = -0.10) with standardized Brier scores; Himmelstein replicated this result in HFC Season 1 (r = -0.15), but not in Season 2 (r = 0.00).…”

Section: Rationale Text Featuresmentioning

confidence: 99%

See 2 more Smart Citations

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein

2023

International Series in Operations Research &Amp; Management Science

View full text Add to dashboard Cite

Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: 1) accuracy-related measures, such as proper scores, modelbased estimates of accuracy and excess volatility scores; 2) intersubjective measures, including proxy, surrogate and similarity scores; 3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; 4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and 5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term prediction and corporate forecasting tournaments.

show abstract

Section: Research Synthesismentioning

confidence: 98%

Section: Behavioralmentioning

confidence: 99%

Section: Rationale Text Featuresmentioning

confidence: 99%

See 1 more Smart Citation

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein

2023

International Series in Operations Research &Amp; Management Science

View full text Add to dashboard Cite

show abstract

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein²

2022

Preprint

View full text Add to dashboard Cite

Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: 1) accuracy-related measures, such as proper scores, model-based estimates of accuracy and excess volatility scores; 2) intersubjective measures, including proxy, surrogate and similarity scores; 3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; 4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and 5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term, existential risk prediction and corporate forecasting tournaments.

show abstract

Measuring Forecasting Skill from Text

Cited by 2 publications

References 47 publications

Talent Spotting in Crowd Prediction

Talent Spotting in Crowd Prediction

Talent Spotting in Crowd Prediction

Contact Info

Product

Resources

About