Settling the robust learnability of mixtures of Gaussians

Abstract

This work represents a natural coalescence of two important lines of work -learning mixtures of Gaussians and algorithmic robust statistics. In particular we give the first provably robust algorithm for learning mixtures of any constant number of Gaussians. We require only mild assumptions on the mixing weights (bounded fractionality) and that the total variation distance between components is bounded away from zero. At the heart of our algorithm is a new method for proving dimension-independent polynomial identifiability through applying a carefully chosen sequence of differential operations to certain generating functions that not only encode the parameters we would like to learn but also the system of polynomial equations we would like to solve. We show how the symbolic identities we derive can be directly used to analyze a natural sum-of-squares relaxation. CCS CONCEPTS• Theory of computation → Unsupervised learning and clustering.

List-decodable covariance estimation

“…This progress in obtaining clustering algorithms for nonspherical mixtures is a key component in the recent resolution of the problem of robust learning of mixtures of arbitrary Gaussians [1,26]. Our list-decodable covariance estimation algorithm immediately upgrades the above results and resolves the problem of finding a 𝑑 poly(𝑘) sample and 𝑛 poly(𝑘) time algorithm for the problem.…”

“…Learning Arbitrary Gaussian Mixtures. Our work is related (but incomparable and complementary, in both results and techniques) to the recent resolution of the problem of robust learning of a mixture of 𝑘-arbitrary Gaussians [1,26]. When viewed from our vantage point, these works give a polynomial time algorithm (for any fixed 𝑘) to learn the parameters of a mixture of 𝑘 Gaussians given an 𝜀-corrupted input sample.…”

“…When viewed from our vantage point, these works give a polynomial time algorithm (for any fixed 𝑘) to learn the parameters of a mixture of 𝑘 Gaussians given an 𝜀-corrupted input sample. The algorithms of [1,26] do not need strong separation assumptions but crucially need that the fraction of outliers is small (at most ∼ exp(−𝑘!)). In that setting, their algorithm recovers estimates of the components that are close (within some 𝜀 𝐹 (𝑘) in [1]) to those of the unknown mixture.…”

“…Indeed, our techniques are significantly (and necessarily so) different from those in [1,26]. In fact, the algorithms in [1,26] use robust clustering algorithms from [3,11] as a first step with their key new algorithmic components coming after the clustering step.…”

“…Indeed, our techniques are significantly (and necessarily so) different from those in [1,26]. In fact, the algorithms in [1,26] use robust clustering algorithms from [3,11] as a first step with their key new algorithmic components coming after the clustering step. We note that using our new list-decodable covariance estimation algorithm in lieu of the clustering algorithm in the first step will somewhat simplify their proof though the bulk of their analysis is to deal with łunclusterablež mixtures.…”