Measuring Forecasting Skill from Text

Abstract

People vary in their ability to make accurate predictions about the future. Prior studies have shown that some individuals can predict the outcome of future events with consistently better accuracy. This leads to a natural question: what makes some forecasters better than others? In this paper we explore connections between the language people use to describe their predictions and their forecasting skill. Datasets from two different forecasting domains are explored: (1) geopolitical forecasts from Good Judgment Open, an online prediction forum and (2) a corpus of company earnings forecasts made by financial analysts. We present a number of linguistic metrics which are computed over text associated with people's predictions about the future including: uncertainty, readability, and emotion. By studying linguistic factors associated with predictions, we are able to shed some light on the approach taken by skilled forecasters. Furthermore, we demonstrate that it is possible to accurately predict forecasting skill using a model that is based solely on language. This could potentially be useful for identifying accurate predictions or potentially skilled forecasters earlier. 1

Sources and Corpora

“…Also, similar backgrounds and knowledge for professionals is no guarantee that their opinions will also be similar: differing analysis methods or information can result in different opinions and in reports with different levels of accuracy. Zong et al [45] order analyst reports by their accuracy in earnings forecasting, and compare the semantic features of the 4,000 most accurate reports with those of the 4,000 most inaccurate reports. They find that the number of uncertain statements, the amount of future temporal orientation, and the number of negative words are significantly associated with inaccurate reports.…”

Organizing Financial Opinions

“…One step in this research direction is to use tailor-made methods and features for financial documents. Although many studies use prediction accuracy as a proxy for the quality of a financial opinion, annotated benchmark datasets are still necessary because even high-quality reports are not always accurate [84]. In Chap.…”

Modeling Language Usage and Listener Engagement in Podcasts

“…Predicting Performance from Language Previous research in natural language processing has explored the connections between textual features and audience engagement in books (Ganjigunte Ashok et al, 2013;Maharjan et al, 2018), YouTube (Kleinberg et al, 2018), news (Naseri and Zamani, 2019), TED talks (Tanveer et al, 2018), and tweets (Tan et al, 2014;Lampos et al, 2014). Other works have modeled the relationship between text and various performance metrics such as movie quote memorability (Danescu-Niculescu-Mizil et al, 2012), forecasting ability (Zong et al, 2020), congressional bill survival (Yano et al, 2012), success of job interviews (Naim et al, 2016), and impact of academic papers (Yogatama et al, 2011;Li et al, 2019), in addition to the entire field of sentiment and opinion mining of data such as user reviews (Pang et al, 2002).…”