We Ran 9 Billion Regressions: Eliminating False Positives through Computational Model Robustness

Abstract

False positive findings are a growing problem in many research literatures. We argue that excessive false positives often stem from model uncertainty. There are many plausible ways of specifying a regression model, but researchers typically report only a few preferred estimates. This raises the concern that such research reveals only a small fraction of the possible results and may easily lead to nonrobust, false positive conclusions. It is often unclear how much the results are driven by model specification and how much the results would change if a different plausible model were used. Computational model robustness analysis addresses this challenge by estimating all possible models from a theoretically informed model space. We use large-scale random noise simulations to show (1) the problem of excess false positive errors under model uncertainty and (2) that computational robustness analysis can identify and eliminate false positives caused by model uncertainty. We also draw on a series of empirical applications to further illustrate issues of model uncertainty and estimate instability. Computational robustness analysis offers a method for relaxing modeling assumptions and improving the transparency of applied research.

Machine Learning for Sociology

“…for example, leading to omitted variable bias in estimation. Similarly, with many inputs, one generally runs the risk of model misspecification (Belloni et al, 2014;Ho et al, 2007;King & Nielsen, 2016;Raftery, 1995;Young & Holsteen, 2017;Muñoz & Young, 2018). Athey & Imbens (2015) develop a measure of sensitivity to misspecification.…”

The multiplicity of analysis strategies jeopardizes replicability: lessons learned across disciplines

“…Common examples of this strategy include extreme bounds analysis in econometrics [80], multimodel ensembles [34] in hydrology and climatology and sensitivity analyses, which are used across many disciplines. More recent approaches to report uncertainty include the "vibration of effects" approach [28], "specification curve analysis" [29], "multiverse analysis" [30] "multimodel analysis" [32] and "compu-tational robustness analysis" [33], as discussed previously. Silberzahn et al [20] go a step further and propose the reporting of the results of different teams of researchers analyzing the same research question on the same data set.…”

“…There are for instance a number of recently proposed approaches which assess the robustness of research findings to alternative analytical pathways by reporting the results of a large number of analysis strategies: the "vibration of effects" approach in epidemiology [28], "specification curve analysis" [29] and "multiverse analysis" in psychology [30], a "measure of robustness to misspecification" in economics [31] or "multimodel analysis" [32] and "computational robustness analysis" [33] in sociology. In other disciplines, including climatology, ecology and risk analysis, there is a long-standing tradition of addressing the robustness to alternative analysis strategies through sensitivity analyses, multimodel ensembles [34] and Bayesian model averaging [35,36].…”

The Crowdsourced Replication Initiative: Investigating Immigration and Social Policy Preferences. Executive Report.

“…Even using the same data and same models we may observe researcher variability. Although some argue for the utility of millions of models (Muñoz and Young 2018), in crowdsourced research the set of models are not a simulation but a set of real models derived from practicing social scientists. No matter how strong the statistical methods, data itself cannot produce the underlying causal model, this requires rational construction of theory, exclusion restrictions, attention to confounding and counterfactuals (Pearl 2010) 5 .…”