Agnostic Estimation of Mean and Covariance

Lai, Kevin A.; Rao, Anup; Vempala, Santosh

doi:10.1109/focs.2016.76

Cited by 142 publications

(171 citation statements)

References 32 publications

Supporting

Mentioning

165

Contrasting

Order By: Relevance

“…Note that a type of "shortest interval" estimator has also been employed in the work on mean estimation for contaminated i.i.d. data [18], but was used as an outlier screening step in that context, rather than a mean estimator. Incidentally, our hybrid estimator to be introduced later will employ a different screening approach based on the median, and then use the shorth estimator to return a more accurate mean estimate.…”

Section: Estimatorsmentioning

confidence: 99%

Mean estimation for entangled single-sample distributions

Pensia

Jog

2019

2019 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

We consider the problem of estimating the common mean of independently sampled data, where samples are drawn in a possibly non-identical manner from symmetric, unimodal distributions with a common mean. This generalizes the setting of Gaussian mixture modeling, since the number of distinct mixture components may diverge with the number of observations. We propose an estimator that adapts to the level of heterogeneity in the data, achieving near-optimality in both the i.i.d. setting and some heterogeneous settings, where the fraction of "low-noise" points is as small as log n n . Our estimator is a hybrid of the modal interval, shorth, and median estimators from classical statistics; however, the key technical contributions rely on novel empirical process theory results that we derive for independent but non-i.i.d. data. In the multivariate setting, we generalize our theory to mean estimation for mixtures of radially symmetric distributions, and derive minimax lower bounds on the expected error of any estimator that is agnostic to the scales of individual data points. Finally, we describe an extension of our estimators applicable to linear regression. In the multivariate mean estimation and regression settings, we present computationally feasible versions of our estimators that run in time polynomial in the number of data points., for any C > 0. Note that these events are dependent and thus we'd use the following lemma, which shows that conditioned on the inclusion of points in each of two disjoint intervals, the distributions of the histograms on each of the intervals behave independently:Bernoulli random variables. We calculate the following quantities required for the Berry-Esseen Theorem,By the Berry-Esseen Theorem [31], we havewhere φ(t) def = P(g ≤ t) and g ∼ N (0, 1). Therefore,

show abstract

Section: Estimatorsmentioning

confidence: 99%

Mean estimation for entangled single-sample distributions

Pensia

Jog

2019

2019 IEEE International Symposium on Information Theory (ISIT)

View full text Add to dashboard Cite

show abstract

“…On the other hand, a number of natural approaches (e.g., naive outlier removal, coordinate-wise median, geometric median, etc.) can only guarantee error Ω(ǫ √ d) (see, e.g., [DKK + 16,LRV16]), even in the infinite sample regime. That is, the performance of these estimators degrades polynomially with the dimension d, which is clearly unacceptable in high dimensions.…”

mentioning

confidence: 99%

“…Recent work [DKK + 16,LRV16] gave the first polynomial time robust estimators for a range of high-dimensional statistical tasks, including mean and covariance estimation. Specifically, [DKK + 16] obtained the first robust estimators for the mean with dimension-independent error guarantees, i.e., whose error only depends on the fraction of corrupted samples ǫ but not on the dimensionality of the data.…”

mentioning

confidence: 99%

“…Specifically, [DKK + 16] obtained the first robust estimators for the mean with dimension-independent error guarantees, i.e., whose error only depends on the fraction of corrupted samples ǫ but not on the dimensionality of the data. Since the dissemination of [DKK + 16, LRV16], there has been a substantial number of subsequent works obtaining robust learning algorithms for a variety of unsupervised and supervised high-dimensional models. (See Section 1.3 for a summary of related work.…”

mentioning

confidence: 99%

“…In addition to its potential practical implications, we believe that understanding the above question is of fundamental theoretical interest as it can elucidate the effect of the robustness requirement on the computational complexity of highdimensional statistical learning/estimation. For example, for the prototypical problem of robustly estimating the mean of a high-dimensional distribution, previous robust algorithms [DKK + 16, LRV16,SCV18] have runtime at least Ω(N d 2 ) for constant ǫ. Since the input size is Θ(N d), we would like to obtain algorithms that run in time O(N d)/ poly(ǫ), where the O(·) notation hides logarithmic factors in its argument.…”

mentioning

confidence: 99%

See 2 more Smart Citations

High-Dimensional Robust Mean Estimation in Nearly-Linear Time

Cheng¹,

Diakonikolas²,

Ge³

2019

Proceedings of the Thirtieth Annual ACM-SIAM Symposium on Discrete Algorithms

132

View full text Add to dashboard Cite

We study the fundamental problem of high-dimensional mean estimation in a robust model where a constant fraction of the samples are adversarially corrupted. Recent work gave the first polynomial time algorithms for this problem with dimension-independent error guarantees for several families of structured distributions.In this work, we give the first nearly-linear time algorithms for high-dimensional robust mean estimation. Specifically, we focus on distributions with (i) known covariance and sub-gaussian tails, and (ii) unknown bounded covariance. Given N samples on R d , an ǫ-fraction of which may be arbitrarily corrupted, our algorithms run in time O(N d)/poly(ǫ) and approximate the true mean within the information-theoretically optimal error, up to constant factors. Previous robust algorithms with comparable error guarantees have running times Ω(N d 2 ), for ǫ = Ω(1).Our algorithms rely on a natural family of SDPs parameterized by our current guess ν for the unknown mean µ ⋆ . We give a win-win analysis establishing the following: either a nearoptimal solution to the primal SDP yields a good candidate for µ ⋆ -independent of our current guess ν -or a near-optimal solution to the dual SDP yields a new guess ν ′ whose distance from µ ⋆ is smaller by a constant factor. We exploit the special structure of the corresponding SDPs to show that they are approximately solvable in nearly-linear time. Our approach is quite general, and we believe it can also be applied to obtain nearly-linear time algorithms for other high-dimensional robust learning problems.

show abstract

Subspace Approximation with Outliers

Deshpande

Pratap

2020

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Agnostic Estimation of Mean and Covariance

Cited by 142 publications

References 32 publications

Mean estimation for entangled single-sample distributions

Mean estimation for entangled single-sample distributions

High-Dimensional Robust Mean Estimation in Nearly-Linear Time

Subspace Approximation with Outliers

Contact Info

Product

Resources

About