Outlier-Robust Clustering of Gaussians and Other Non-Spherical Mixtures

Bakshi, Ainesh; Diakonikolas, Ilias; Hopkins, Samuel B.; Kane, Daniel M.; Karmalkar, Sushrut; Kothari, Pravesh K.

doi:10.1109/focs46700.2020.00023

Cited by 15 publications

(51 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Unfortunately this strategy inherently introduces polynomial factors in 𝑑 and it cannot give what we are after. For the special case of clusterable mixtures of Gaussians, Bakshi and Kothari [5] and Diakonikolas et al [20] proved dimension independent polynomial identifiability and their approach was based on classifying the ways in which two single Gaussians can have total variation distance close to one. When it comes to the more general problem of handling mixtures where the components can overlap non-trivially it seems difficult to follow the same route because we can no longer match components from the two mixtures to each other and almost cancel them both out.…”

Section: Key Challengesmentioning

confidence: 99%

“…Diakonikolas et al [21] gave a robust algorithm for learning mixtures of spherical Gaussians. In recent breakthroughs Bakshi and Kothari [5] and Diakonikolas et al [20] gave a robust algorithm for learning clusterable mixtures of Gaussians and building on this Kane [37] gave a robust algorithm for learning mixtures of two Gaussians. We note that these works do place some mild restrictions on the mixing weights and the variances.…”

mentioning

confidence: 99%

“…The algorithms of Bakshi and Kothari [5] and Diakonikolas et al [20] rely on the powerful sum-of-squares hierarchy [45]. One view is that it finds an operator, called the pseudo-expectation, that maps low degree polynomials to real values.…”

mentioning

confidence: 99%

See 2 more Smart Citations

Settling the robust learnability of mixtures of Gaussians

Liu

Moitra

2021

Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing

View full text Add to dashboard Cite

This work represents a natural coalescence of two important lines of work -learning mixtures of Gaussians and algorithmic robust statistics. In particular we give the first provably robust algorithm for learning mixtures of any constant number of Gaussians. We require only mild assumptions on the mixing weights (bounded fractionality) and that the total variation distance between components is bounded away from zero. At the heart of our algorithm is a new method for proving dimension-independent polynomial identifiability through applying a carefully chosen sequence of differential operations to certain generating functions that not only encode the parameters we would like to learn but also the system of polynomial equations we would like to solve. We show how the symbolic identities we derive can be directly used to analyze a natural sum-of-squares relaxation. CCS CONCEPTS• Theory of computation → Unsupervised learning and clustering.

show abstract

Section: Key Challengesmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Settling the robust learnability of mixtures of Gaussians

Liu

Moitra

2021

Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing

View full text Add to dashboard Cite

show abstract

“…Furthermore, our lower bound implies that, for the class of distributions we consider, if a pair of distributions is close in variational distance, then the distributions also have close parameters. This type of implication is integral to certain arguments in outlier-robust moment estimation algorithms and clustering [4,20].…”

mentioning

confidence: 99%

“…The functional form of the TV distance bound is often much more useful in practice because it can be directly evaluated based on only the means and covariances of the distribution. This has opened up the door for new applications to a variety of areas, such as analyzing ReLU networks [34], distribution learning [3,4], private distribution testing [7,8], and average-case reductions [6].…”

mentioning

confidence: 99%

Lower Bounds on the Total Variation Distance Between Mixtures of Two Gaussians

Davies¹,

Mazumdar²,

Pal³

et al. 2021

Preprint

View full text Add to dashboard Cite

Mixtures of high dimensional Gaussian distributions have been studied extensively in statistics and learning theory. While the total variation distance appears naturally in the sample complexity of distribution learning, it is analytically difficult to obtain tight lower bounds for mixtures. Exploiting a connection between total variation distance and the characteristic function of the mixture, we provide fairly tight functional approximations. This enables us to derive new lower bounds on the total variation distance between pairs of two-component Gaussian mixtures that have a shared covariance matrix. IntroductionLet N (µ, Σ) denote the d-dimensional Gaussian distribution with mean µ ∈ R d and positive definite covariance matrix, where w i ∈ R + with k i=1 w i = 1 are the mixing weights, µ i ∈ R d are the means, and Σ i ∈ R d×d are the covariance matrices. Mixtures of Gaussian distributions have been studied intensively due to broad applicability to statistical problems [2,10,11,21,22,28,29,31,32].The variational distance (a.k.a., the total variation (TV) distance) between two distributions f, f with same sample space Ω and sigma algebra S is defined as follows:

show abstract

List-decodable covariance estimation

Ivkov

Kothari

2022

Proceedings of the 54th Annual ACM SIGACT Symposium on Theory of Computing

Self Cite

View full text Add to dashboard Cite

We give the first polynomial time algorithm for list-decodable covariance estimation. For any 𝛼 > 0, our algorithm takes input a sample 𝑌 ⊆ ℝ 𝑑 of size 𝑛 ⩾ 𝑑 poly(1/𝛼) obtained by adversarially corrupting an (1 − 𝛼)𝑛 points in an i.i.d. sample 𝑋 of size 𝑛 from the Gaussian distribution with unknown mean 𝜇 * and covariance Σ * . In 𝑛 poly(1/𝛼) time, it outputs a constant-size list of 𝑘 = 𝑘 (𝛼) = (1/𝛼) poly(1/𝛼) candidate parameters that, with high probability, contains a ( μ, Σ) such that the total variation distance 𝑇𝑉 (N (𝜇 * , Σ * ), N ( μ, Σ)) < 1−𝑂 𝛼 (1). This is a statistically strongest notion of distance and implies multiplicative spectral and relative Frobenius distance approximation with dimension independent error. Our algorithm works more generally for any distribution 𝐷 that possesses low-degree sum-of-squares certificates of two natural analytic properties: 1) anti-concentration of one-dimensional marginals and 2) hypercontractivity of degree 2 polynomials.Prior to our work, the only known results for estimating covariance in the list-decodable setting were for the special cases of list-decodable linear regression and subspace recovery [Karmalkar-Klivans-Kothari 2019. Even for these special cases, the known error guarantees are weak and in particular, the algorithms need super-polynomial time for any sub-constant (in dimension 𝑑) target error in natural norms. Our result, as a corollary, yields the first polynomial time exact algorithm for list-decodable linear regression and subspace recovery that, in particular, obtain 2 − poly(𝑑) error in polynomial-time in the underlying dimension. CCS CONCEPTS• Theory of computation → Design and analysis of algorithms; Complexity theory and logic.

show abstract

Outlier-Robust Clustering of Gaussians and Other Non-Spherical Mixtures

Cited by 15 publications

References 25 publications

Settling the robust learnability of mixtures of Gaussians

Settling the robust learnability of mixtures of Gaussians

Lower Bounds on the Total Variation Distance Between Mixtures of Two Gaussians

List-decodable covariance estimation

Contact Info

Product

Resources

About