Near-optimal Sample Complexity Bounds for Robust Learning of Gaussian Mixtures via Compression Schemes

Abstract

We introduce a novel technique for distribution learning based on a notion of sample compression . Any class of distributions that allows such a compression scheme can be learned with few samples. Moreover, if a class of distributions has such a compression scheme, then so do the classes of products and mixtures of those distributions. As an application of this technique, we prove that ˜Θ( kd 2 /ε 2 ) samples are necessary and sufficient for learning a mixture of k Gaussians in R d , up to error ε in total variation distance. This improves both the known upper bounds and lower bounds for this problem. For mixtures of axis-aligned Gaussians, we show that Õ( kd /ε 2 ) samples suffice, matching a known lower bound. Moreover, these results hold in an agnostic learning (or robust estimation) setting, in which the target distribution is only approximately a mixture of Gaussians. Our main upper bound is proven by showing that the class of Gaussians in R d admits a small compression scheme.

Solar PV Inverter Reactive Power Disaggregation and Control Setting Estimation

“…Note that an advantage of this method is that we can quantify the minimum amount of training samples M ′ that we need so that the total variation distance from the true distribution is no more than ϵ. Bounds for the minimum number of samples are available in [39]. In our case, this bound is…”

DISCA: high-throughput cryo-ET structural pattern mining by deep unsupervised clustering

“…This leads to the following theorem. In terms of sample complexity, we leverage work by [42] that has shown thatΘ(KP 2 /ε 2 ) samples are both necessary and sufficient for learning mixed multivariate Gaussian distributions with K components in a P-dimensional feature space with up to ε error in total variation distance. This result implies that learning reasonably accurate models that achieve a low, constant error ε requires rel-atively few samples in practice, as K and P are assumed to be small compared to N in large-scale datasets.…”

Generative Continual Concept Learning

“…The second term denotes the distance between the task distribution and the fitted GMM. When the PDP hypothesis holds and the model learns a task well, this term is small as we can approximate φ(q (t) ) withp Ashtiani et al (Ashtiani et al 2018) for a rigorous analysis of estimating a distribution with GMM). In other words, this term is small if the classes are learned as concepts.…”