High-Dimensional Robust Mean Estimation in Nearly-Linear Time

Abstract

We study the fundamental problem of high-dimensional mean estimation in a robust model where a constant fraction of the samples are adversarially corrupted. Recent work gave the first polynomial time algorithms for this problem with dimension-independent error guarantees for several families of structured distributions.In this work, we give the first nearly-linear time algorithms for high-dimensional robust mean estimation. Specifically, we focus on distributions with (i) known covariance and sub-gaussian tails, and (ii) unknown bounded covariance. Given N samples on R d , an ǫ-fraction of which may be arbitrarily corrupted, our algorithms run in time O(N d)/poly(ǫ) and approximate the true mean within the information-theoretically optimal error, up to constant factors. Previous robust algorithms with comparable error guarantees have running times Ω(N d 2 ), for ǫ = Ω(1).Our algorithms rely on a natural family of SDPs parameterized by our current guess ν for the unknown mean µ ⋆ . We give a win-win analysis establishing the following: either a nearoptimal solution to the primal SDP yields a good candidate for µ ⋆ -independent of our current guess ν -or a near-optimal solution to the dual SDP yields a new guess ν ′ whose distance from µ ⋆ is smaller by a constant factor. We exploit the special structure of the corresponding SDPs to show that they are approximately solvable in nearly-linear time. Our approach is quite general, and we believe it can also be applied to obtain nearly-linear time algorithms for other high-dimensional robust learning problems.

ERM and RERM are optimal estimators for regression problems when malicious outliers corrupt the labels

“…In [22], the authors focused on the problem of high-dimensional linear regression in a robust model where an ε-fraction of the samples can be adversarially corrupted. Robust regression problems have also been studied in [15,21,38,6]. Above-mentioned articles assume corruption both in the design and the label.…”

Heavy-tailed Streaming Statistical Estimation

“…This guarantee is still sub-optimal compared to the optimal sub-Gaussian rate [35]. Existing polynomial time algorithms having optimal performance are for the batch setting and require either storing the entire dataset [6,5,36,32,11] or have O(p log(1/δ)) storage complexity [8]. On the other hand, we argue that trading off some statistical accuracy for a large savings in memory and computation is favorable in practice.…”

“…In the batch-setting, Hopkins [27] proposed a robust mean estimator that can be computed in polynomial time and which matches the error guarantees achieved by the empirical mean on Gaussian data. After this work, efficient algorithms with improved asymptotic runtimes were given in [10,6,5,32,8]. Related works [43,38] provide more practical and computationally-efficient algorithms with slightly sub-optimal rates.…”

“…We consider the linear regression model described in Eq. (5). Assume that the covariates x ∈ R p have bounded 4 th moments and a non-singular covariance matrix Σ with bounded operator norm, and the noise w has bounded 2 nd moments.…”

“…There has also been a burgeoning line of recent work that relaxes these assumptions and allows for heavy-tailed underlying distributions [3,28,35], but the resulting algorithms are often not only complex, but are also specifically batch learning algorithms that require the entire dataset, which limits their scalability. For instance, many popular polynomial time algorithms on heavy-tailed mean estimation [9,43,5,44,6,32] and heavy-tailed linear regression [28,46,42,38] need to store the entire dataset. On the other hand, most successful practical modern learning algorithms are iterative, light-weight and access data in a "streaming" fashion.…”