Provably Auditing Ordinary Least Squares in Low Dimensions

Abstract

Measuring the stability of conclusions derived from Ordinary Least Squares linear regression is critically important, but most metrics either only measure local stability (i.e. against infinitesimal changes in the data), or are only interpretable under statistical assumptions. Recent work proposes a simple, global, finite-sample stability metric: the minimum number of samples that need to be removed so that rerunning the analysis overturns the conclusion [BGM20], specifically meaning that the sign of a particular coefficient of the estimated regressor changes. However, besides the trivial exponential-time algorithm, the only approach for computing this metric is a greedy heuristic that lacks provable guarantees under reasonable, verifiable assumptions; the heuristic provides a loose upper bound on the stability and also cannot certify lower bounds on it.We show that in the low-dimensional regime where the number of covariates is a constant but the number of samples is large, there are efficient algorithms for provably estimating (a fractional version of) this metric. Applying our algorithms to the Boston Housing dataset, we exhibit regression analyses where we can estimate the stability up to a factor of 3 better than the greedy heuristic, and analyses where we can certify stability to dropping even a majority of the samples.

Sample Fit Reliability

“…) involved in empirical research, we aspire to shift the attention in applied econometrics to the importance of data reliability, much as the "credibility revolution" (Angrist & Pischke, 2010) shifted attention to the importance of research design. With the development of SFR, we provide one approach for assessing sample fit, but there also are complementary approaches, such as recently proposed methods by Broderick et al (2020), Kuschnig et al (2021) and Moitra and Rohatgi (2022) that challenge results in an adversarial manner.…”

“…Our work relates to several recent studies that address aspects of sample fit. Works by Broderick, Giordano, and Meager (2020), Kuschnig et al (2021) and Moitra and Rohatgi (2022) identify data points within a sample that distort the estimation of model parameters when they are removed from an analysis in an adversarial manner. For example, Broderick et al (2020) develop a computationally-efficient method to approximately identify the maximum influence perturbation (AMIP).…”

“…Kuschnig et al (2021) further adapt perturbation testing to cases where jointly influential data points are a concern (i.e. masking, see Rousseeuw & Van Zomeren, 1990), and Moitra and Rohatgi (2022) provide theoretical guarantees for low dimensional cases.…”