SAGE: A Hybrid Geopolitical Event Forecasting System

Galstyan, Aram; Satyukov, Gleb; Benjamin, D. M.; Abeliuk, Andrés; Mirtaheri, Mehrnoosh; Hossain, K. S. M. Tozammel; Szekely, Pedro; Ferrara, Emilio; Matsui, Akira; Steyvers, Mark; Bennet, Stephen; Budescu, David V.; Himmelstein, Mark; Ward, Michael D.; Beger, Andreas; Catasta, Michele; Sosič, Rok; Leskovec, Jure; Atanasov, Pavel; Regina, Joseph; Sethi, Rajiv; Abbas, Ali E.

doi:10.24963/ijcai.2019/955

Cited by 20 publications

(18 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this work, we study forecasts about geopolitical events. These forecasts are created on a hybrid forecasting platform 29 , Synergistic Anticipation of Geopolitical Events (SAGE), designed for this purpose. One of the key innovations of SAGE is that it allows forecasters to interact with computer-generated output during their process of generating forecasts.…”

Section: Advice Takingmentioning

confidence: 99%

Quantifying machine influence over human forecasters

Abeliuk

Benjamin

Galstyan

2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

Crowdsourcing human forecasts and machine learning models each show promise in predicting future geopolitical outcomes. Crowdsourcing increases accuracy by pooling knowledge, which mitigates individual errors. On the other hand, advances in machine learning have led to machine models that increase accuracy due to their ability to parameterize and adapt to changing environments. To capitalize on the unique advantages of each method, recent efforts have shown improvements by “hybridizing” forecasts—pairing human forecasters with machine models. This study analyzes the effectiveness of such a hybrid system. In a perfect world, independent reasoning by the forecasters combined with the analytic capabilities of the machine models should complement each other to arrive at an ultimately more accurate forecast. However, well-documented biases describe how humans often mistrust and under-utilize such models in their forecasts. In this work, we present a model that can be used to estimate the trust that humans assign to a machine. We use forecasts made in the absence of machine models as prior beliefs to quantify the weights placed on the models. Our model can be used to uncover other aspects of forecasters’ decision-making processes. We find that forecasters trust the model rarely, in a pattern that suggests they treat machine models similarly to expert advisors, but only the best forecasters trust the models when they can be expected to perform well. We also find that forecasters tend to choose models that conform to their prior beliefs as opposed to anchoring on the model forecast. Our results suggest machine models can improve the judgment of a human pool but highlight the importance of accounting for trust and cognitive biases involved in the human judgment process.

show abstract

Section: Advice Takingmentioning

confidence: 99%

Quantifying machine influence over human forecasters

Abeliuk

Benjamin

Galstyan

2020

Sci Rep

Self Cite

View full text Add to dashboard Cite

show abstract

“…Though there are large differences in the designs of these studies, they are often interpreted as yielding conflicting results (Hou & Jung, 2021), and several recent reviews attempt to synthesize these findings in detail (Burton et al, 2020;Mahmud et al, 2022). This issue is of particular relevance to applications of hybrid forecasting, a topic recently explored in a large scale, longitudinal geopolitical forecasting contest known as the Hybrid Forecasting Competition (HFC) (Himmelstein et al, 2021;Morstatter et al, 2019;Zellner et al, 2021). The goal of the HFC was to combine human judgment and algorithmic forecasts to produce a superior alternative to either source of information on its own.…”

Section: Introductionmentioning

confidence: 99%

“…They can consult with experts they believe to be better informed than they are, larger groups of people to obtain a consensus, and increasingly even sophisticated statistical models which provide objective estimates based on specified criteria. This is particularly pertinent in the domain of forecasting economic and political events, such as those used in geopolitical forecasting tournaments (Himmelstein et al, 2021; Mellers et al, 2015; Morstatter et al, 2019; Tetlock & Gardner, 2016), which can be highly specialized and idiosyncratic. Individuals may feel comfortable relying on their own knowledge to forecast the outcome of a local election but not when asked to predict election results in a country on the other side of the world.…”

Section: Introductionmentioning

confidence: 99%

Preference for human or algorithmic forecasting advice does not predict if and how it is used

Himmelstein

Budescu

2022

Behavioral Decision Making

Self Cite

View full text Add to dashboard Cite

Past research has found that people treat advice differently depending on its source.In many cases, people seem to prefer human advice to algorithms, but in others, there is a reversal, and people seem to prefer algorithmic advice. Across two studies, we examine the persuasiveness of, and judges' preferences for, advice from different sources when forecasting geopolitical events. We find that judges report domainspecific preferences, preferring human advice in the domain of politics and algorithmic advice in the domain of economics. In Study 2, participants report a preference for hybrid advice, that combines human and algorithmic sources, to either one on it's own regardless of domain. More importantly, we find that these preferences did not affect persuasiveness of advice from these different sources, regardless of domain.Judges were primarily sensitive to quantitative features pertaining to the similarity between their initial beliefs and the advice they were offered, such as the distance between them and the relative advisor confidence, when deciding whether to revise their initial beliefs in light of advice, rather than the source that generated the advice.

show abstract

“…One approach could learn from highly structured eventcoded data such as ICEWS (Boschee et al, 2015) and GDELT (Leetaru and Schrodt, 2013). When these datasets are used for forecasting, they are often represented as a time series (Morstatter et al, 2019;Ramakrishnan et al, 2014b), in which each data point is associated with a timestamp. Another approach is script-learning, in which a model is provided with a chain of events and a subsequent event and is asked to predict the relation between the chain and the "future" event (Hu et al, 2017;Li et al, 2018;Lv et al, 2019).…”

Section: Related Workmentioning

confidence: 99%

ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data

Jin¹,

Khanna²,

Kim³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

Self Cite

View full text Add to dashboard Cite

Event forecasting is a challenging, yet important task, as humans seek to constantly plan for the future. Existing automated forecasting studies rely mostly on structured data, such as time-series or event-based knowledge graphs, to help predict future events. In this work, we aim to formulate a task, construct a dataset, and provide benchmarks for developing methods for event forecasting with large volumes of unstructured text data. To simulate the forecasting scenario on temporal news documents, we formulate the problem as a restricted-domain, multiple-choice, question-answering (QA) task. Unlike existing QA tasks, our task limits accessible information, and thus a model has to make a forecasting judgement. To showcase the usefulness of this task formulation, we introduce FORECASTQA, a question-answering dataset consisting of 10,392 event forecasting questions, which have been collected and verified via crowdsourcing efforts. We present our experiments on FORECASTQA using BERTbased models and find that our best model achieves 61.0% accuracy on the dataset, which still lags behind human performance by about 19%. We hope FORECASTQA will support future research efforts in bridging this gap. 1 1 https://inklab.usc.edu/ForecastQA/ A) South Korea [0.41] B) Syria [0.28] C) South Africa [0.15] D) Portugal [0.16] Yes [0.38] / No [0.62] Q: Who will drop Japan as a trading partner in August 2019?(1/1/19) Apart from the fact of being one another's closest neighbors, the people of South Korea and Japan have a remarkable amount in common. Economically, they are among one another's biggest trading partners. And yet, time and again, relations between Seoul and Tokyo are marked, not by mutual support and co-operation but by anger, reproach and exasperation. Q: Will primary schools in Europe admit non-vaccinated children around September 2019?(3/8/18) Public officials and health experts had given several warnings: Do not allow a student in school if they had not been vaccinated against measles.(6/27/19) Fines for parents refusing measles jab. Parents will be fined up to € 2,500 if they don't vaccinate their children against measles under draft legislation in Germany which also threatens exclusion from crèches, nurseries and schools.earlier than timestamp (2019-08-01) earlier than timestamp

show abstract

SAGE: A Hybrid Geopolitical Event Forecasting System

Cited by 20 publications

References 1 publication

Quantifying machine influence over human forecasters

Quantifying machine influence over human forecasters

Preference for human or algorithmic forecasting advice does not predict if and how it is used

ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data

Contact Info

Product

Resources

About