High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity

Abstract

Although the standard formulations of prediction problems involve fully-observed and noiseless data drawn in an i.i.d. manner, many applications involve noisy and/or missing data, possibly involving dependence, as well. We study these issues in the context of high-dimensional sparse linear regression, and propose novel estimators for the cases of noisy, missing and/or dependent data. Many standard approaches to noisy or missing data, such as those using the EM algorithm, lead to optimization problems that are inherently nonconvex, and it is difficult to establish theoretical guarantees on practical algorithms. While our approach also involves optimizing nonconvex programs, we are able to both analyze the statistical error associated with any global optimum, and more surprisingly, to prove that a simple algorithm based on projected gradient descent will converge in polynomial time to a small neighborhood of the set of all global minimizers. On the statistical side, we provide nonasymptotic bounds that hold with high probability for the cases of noisy, missing and/or dependent data. On the computational side, we prove that under the same types of conditions required for statistical consistency, the projected gradient descent algorithm is guaranteed to converge at a geometric rate to a near-global minimizer. We illustrate these theoretical predictions with simulations, showing close agreement with the predicted scalings. . This reprint differs from the original in pagination and typographic detail. 1 2 P.-L. LOH AND M. J. WAINWRIGHTalistic for many applications, in which covariates may be observed only partially, observed subject to corruption or exhibit some type of dependency.Consider the problem of modeling the voting behavior of politicians: in this setting, votes may be missing due to abstentions, and temporally dependent due to collusion or "tit-for-tat" behavior. Similarly, surveys often suffer from the missing data problem, since users fail to respond to all questions. Sensor network data also tends to be both noisy due to measurement error, and partially missing due to failures or drop-outs of sensors.There are a variety of methods for dealing with noisy and/or missing data, including various heuristic methods, as well as likelihood-based methods involving the expectation-maximization (EM) algorithm (e.g., see the book [8] and references therein). A challenge in this context is the possible nonconvexity of associated optimization problems. For instance, in applications of EM, problems in which the negative likelihood is a convex function often become nonconvex with missing or noisy data. Consequently, although the EM algorithm will converge to a local minimum, it is difficult to guarantee that the local optimum is close to a global minimum.In this paper, we study these issues in the context of high-dimensional sparse linear regression-in particular, in the case when the predictors or covariates are noisy, missing, and/or dependent. Our main contribution is to develop and study simple methods for handling th...

Sparse linear models andl1-regularized 2SLS with high-dimensional endogenous regressors and instruments

“…This result is essentially shown in Loh and Wainwright (2012), Lemma 1, which is a consequence of their Lemmas 12, 13 and 15. Below we re-phrase the argument in their Lemma 1 to give the readers more guidance.…”

“…For a random variable V , as in Vershynin (2012), we define the "sub-Gaussian" norm |V | Ψ := Loh and Wainwright, 2012;Negahban, et. al 2012;Rosenbaum and Tsybakov, 2013).…”

“…The bound in Assumption 2.3 can be derived under lower level conditions (see Lemma B.2, which is a consequence of Lemmas 12, 13 and 15 in Loh and Wainwright, 2012).…”

“…where the second inequality follows fromc ≥ Bound (60) follows exactly the same argument with only one difference: instead of using bound (70) in Lemma 13 of Loh and Wainwright (2012), bound (71) is used and λ max (Σ x ) is replaced withκ.…”

New Theory and Algorithms for Scalable Data Fusion

“…D 2012, now an assistant professor at University of Wisconsin, Madison), Alekh Agarwal (Ph.D. 2012, now research scientist at Microsoft Research), and Po-Ling Loh (Ph.D. expected in 2014). The grant has lead to the refereed conference papers [26,24,33,13], as well as the journal publications [36,2,1,34,14,37,35,25]. …”