Do the robot

Abstract

Increasingly, scholars interested in understanding conflict processes have turned to evaluating out-of-sample forecasts to judge and compare the usefulness of their models. Research in this vein has made significant progress in identifying and avoiding the problem of overfitting sample data. Yet there has been less research providing strategies and tools to practically improve the out-of-sample performance of existing models and connect forecasting improvement to the goal of theory development in conflict studies. In this article, we fill this void by building on lessons from machine learning research. We highlight a set of iterative tasks, which David Blei terms ‘Box’s loop’, that can be summarized as build, compute, critique, and think. While the initial steps of Box’s loop will be familiar to researchers, the underutilized process of model criticism allows researchers to iteratively learn more useful representations of the data generation process from the discrepancies between the trained model and held-out data. To benefit from iterative model criticism, we advise researchers not only to split their available data into separate training and test sets, but also sample from their training data to allow for iterative model development, as is common in machine learning applications. Since practical tools for model criticism in particular are underdeveloped, we also provide software for new visualizations that build upon already existing tools. We use models of civil war onset to provide an illustration of how our machine learning-inspired research design can simultaneously improve out-of-sample forecasting performance and identify useful theoretical contributions. We believe these research strategies can complement existing designs to accelerate innovations across conflict processes.

A review of climate security risk assessment tools

“…The ground truth can be retrieved via interviews with regional experts, media posts, and impacts of climatic events observed in quantitative data, to name a few examples (Bakkensen et al, 2017; Busby et al, 2018; Mach & Kraan, 2021; Visser et al, 2020). In this sense, the quantitative predictive tools can draw on out-of-sample heuristics, to evaluate the model performance on a test set not used for the model construction (Busby, 2018; Colaresi & Mahmood, 2017; Hegre et al, 2021). This could be advanced by feeding the model with new observations over time and validating against these new data points.…”

“…All predictive tools conduct some out-of-sample predictions to validate the model performance (Ward & Beger, 2017). In this regard, ViEWS explicitly follows the guidelines by Colaresi and Mahmood (2017). WPS further benchmarks its model against others, including ViEWS (Kuzma et al, 2020).…”

Forecasting Social Conflicts in Africa Using an Epidemic Type Aftershock Sequence Model

“…Recently, various studies incorporating machine learning techniques, like for example [5][6][7], have indicated that certain machine learning methods can deliver accurate forecasts for social conflicts.…”

Revisiting the Contested Role of Natural Resources in Violent Conflict Risk through Machine Learning

“…Ward, Greenhill, and Bakke [45] demonstrate that out-of-sample validation of predictive models is a useful heuristic tool for evaluating causal claims and policy guidance. Colaresi and Mahmood [46] reason that machine learning predictions of conflict can inform theory through the analysis of patterns in the data, mainly discrepancies between observed and modelled outcomes, allowing for nonlinearity and complex interactions between variables.…”