List Decodable Mean Estimation in Nearly Linear Time

Cherapanamjeri, Yeshwanth; Mohanty, Sidhanth; Yau, Morris

doi:10.1109/focs46700.2020.00022

Cited by 8 publications

(28 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The concurrent work [CMY20] gives an SDP-based algorithm whose runtime is Õ(nd)/poly(α), i.e., near-optimal as a function of the dimension d, but suboptimal (by a polynomial factor) as a function of 1/α. We note that the dependence on 1/α is equally significant in some of the key applications of list-decodable learning (e.g., in learning mixture models).…”

Section: Discussionmentioning

confidence: 99%

“…Concurrent and Independent Work. Contemporaneous work [CMY20], using different techniques, gave an algorithm for the same problem with asymptotic running time Õ(nd/α c ), for some (unspecified) constant c. At a high-level, the algorithm of [CMY20] builds on the convex optimization frameworks of [DKK + 16, CDG18], leveraging faster algorithms for solving structured SDPs.…”

Section: Our Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

List-Decodable Mean Estimation via Iterative Multi-Filtering

Diakonikolas,

Kane,

Kongsgaard

2020

Preprint

View full text Add to dashboard Cite

We study the problem of list-decodable mean estimation for bounded covariance distributions. Specifically, we are given a set T of points in R d with the promise that an unknown α-fraction of points in T , where 0 < α < 1/2, are drawn from an unknown mean and bounded covariance distribution D, and no assumptions are made on the remaining points. The goal is to output a small list of hypothesis vectors such that at least one of them is close to the mean of D. We give the first practically viable estimator for this problem. In more detail, our algorithm is sample and computationally efficient, and achieves information-theoretically near-optimal error. While the only prior algorithm for this setting inherently relied on the ellipsoid method, our algorithm is iterative and only uses spectral techniques. Our main technical innovation is the design of a soft outlier removal procedure for high-dimensional heavy-tailed datasets with a majority of outliers. * Authors are in alphabetical order.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Our Contributionsmentioning

confidence: 99%

List-Decodable Mean Estimation via Iterative Multi-Filtering

Diakonikolas,

Kane,

Kongsgaard

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…For systematic discussions on the basic problems in robust statistics, we refer to Huber (2004) and Hampel et al (2011). Recently, there is a resurgent interest in developing polynomial time robust estimators achieving near optimal recovery guarantees for many parametric estimation problems including mean estimation, linear regression and generalized linear models, where an analyst is given access to samples, in which a fraction of the samples may have been adversarially corrupted, see for example, Cherapanamjeri et al (2020), Jambulapati et al (2021) and the references therein.…”

Section: Introduction Consider a Nonparametric Regression Modelmentioning

confidence: 99%

Robust Nonparametric Regression with Deep Neural Networks

Shen¹,

Jiao²,

Lin³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we study the properties of robust nonparametric estimation using deep neural networks for regression models with heavy tailed error distributions. We establish the non-asymptotic error bounds for a class of robust nonparametric regression estimators using deep neural networks with ReLU activation under suitable smoothness conditions on the regression function and mild conditions on the error term. In particular, we only assume that the error distribution has a finite p-th moment with p greater than one. We also show that the deep robust regression estimators are able to circumvent the curse of dimensionality when the distribution of the predictor is supported on an approximate lower-dimensional set. An important feature of our error bound is that, for ReLU neural networks with network width and network size (number of parameters) no more than the order of the square of the dimensionality d of the predictor, our excess risk bounds depend sub-linearly on the d. Our assumption relaxes the exact manifold support assumption, which could be restrictive and unrealistic in practice. We also relax several crucial assumptions on the data distribution, the target regression function and the neural networks required in the recent literature. Our simulation studies demonstrate that the robust methods can significantly outperform the least squares method when the errors have heavy-tailed distributions and illustrate that the choice of loss function is important in the context of deep nonparametric regression.

show abstract

“…While this runtime dramatically improves over the runtime in [CSV17], the quadratic dependence on n is not ideal in very high-dimensional problem settings. Concurrently to [DKK20], the work [CMY20] proposes a different, descent-based algorithm based on (approximate) positive semidefinite programming, achieving optimal error (up to constant factors) in time O(ndα −C ) for some constant C ≥ 6. When α = Θ(1), the [CMY20] runtime is nearly-linear in the problem input size.…”

Section: Introductionmentioning

confidence: 99%

“…This runtime matches up to logarithmic factors the cost of performing a single k-PCA on the data, which is a natural bottleneck of known algorithms for (very) special cases of our problem, such as clustering wellseparated mixtures. Prior to our work, the fastest list-decodable mean estimation algorithms had runtimes O(n 2 dk 2 ) [DKK20], and O(ndk C ) [CMY20] for an unspecified constant C ≥ 6.…”

mentioning

confidence: 99%