A Theory of Statistical Inference for Ensuring the Robustness of Scientific Results

Abstract

Inference is the process of using facts we know to learn about facts we do not know. A theory of inference gives assumptions necessary to get from the former to the latter, along with a definition for and summary of the resulting uncertainty. Any one theory of inference is neither right nor wrong but merely an axiom that may or may not be useful. Each of the many diverse theories of inference can be valuable for certain applications. However, no existing theory of inference addresses the tendency to choose, from the range of plausible data analysis specifications consistent with prior evidence, those that inadvertently favor one’s own hypotheses. Because the biases from these choices are a growing concern across scientific fields, and in a sense the reason the scientific community was invented in the first place, we introduce a new theory of inference designed to address this critical problem. We introduce hacking intervals, which are the range of a summary statistic one may obtain given a class of possible endogenous manipulations of the data. Hacking intervals require no appeal to hypothetical data sets drawn from imaginary superpopulations. A scientific result with a small hacking interval is more robust to researcher manipulation than one with a larger interval and is often easier to interpret than a classical confidence interval. Some versions of hacking intervals turn out to be equivalent to classical confidence intervals, which means they may also provide a more intuitive and potentially more useful interpretation of classical confidence intervals. This paper was accepted by J. George Shanthikumar, big data analytics.

Visualizing the Implicit Model Selection Tradeoff

“…To demonstrate the advantages of our methods, we train classification and regression models that predict risk, and then apply our methods to compare the resulting models and show how our methods highlight certain issues. Specifically, we train classification and regression models using the dataset provided by Coker, Rudin, and King (2021), which we further preprocess by deleting and discretizing certain features. The dataset contains 6215 observations, each representing a defendant, and contains 14 features which include defendant's gender, age (18-20, 21-22, 23-25, 26-45, and > 45), a binary indicator of whether the most recent charge prior to filling the questionnaire is a felony, a binary indicator of juvenile felonies, binary indicator of juvenile misdemeanors, a binary indicator of juvenile crimes, and the number of prior charges (0, 1, 2-3, and > 3).…”

Exploring the Whole Rashomon Set of Sparse Decision Trees

“…[5,6]. They have been used for decision making [7] and robustness of estimation [8]. Semenova et al [1] show that when the Rashomon set is large, models with various important properties such as interpretability and fairness can exist inside it.…”

Matching Bounds: How Choice of Matching Algorithm Impacts Treatment Effects Estimates and What to Do about It

“…Researcher degrees of freedom There is also one fundamental non-statistical factor that influences the issue presented here: researcher degrees of freedom inherent in choice of matching procedure and hyperparameter values associated with it (Gelman and Loken, 2013;Coker et al, 2018). There is often little to no guidance on how to choose either a 5 See the supplement for a derivation.…”