Sketching for large-scale learning of mixture models

Abstract

Learning parameters from voluminous data can be prohibitive in terms of memory and computational requirements. We propose a "compressive learning" framework where we estimate model parameters from a sketch of the training data. This sketch is a collection of generalized moments of the underlying probability distribution of the data. It can be computed in a single pass on the training set, and is easily computable on streams or distributed datasets. The proposed framework shares similarities with compressive sensing, which aims at drastically reducing the dimension of high-dimensional signals while preserving the ability to reconstruct them. To perform the estimation task, we derive an iterative algorithm analogous to sparse reconstruction algorithms in the context of linear inverse problems. We exemplify our framework with the compressive estimation of a Gaussian Mixture Model (GMM), providing heuristics on the choice of the sketching procedure and theoretical guarantees of reconstruction. We experimentally show on synthetic data that the proposed algorithm yields results comparable to the classical Expectation-Maximization (EM) technique while requiring significantly less memory and fewer computations when the number of database elements is large. We further demonstrate the potential of the approach on real large-scale data (over 10 8 training samples) for the task of model-based speaker verification.Finally, we draw some connections between the proposed framework and approximate Hilbert space embedding of probability distributions using random features. We show that the proposed sketching operator can be seen as an innovative method to design translation-invariant kernels adapted to the analysis of GMMs. We also use this theoretical framework to derive information preservation guarantees, in the spirit of infinite-dimensional compressive sensing.

Sketched clustering via hybrid approximate message passing

“…To recover the centroids C from y, the state-of-the-art algorithm is compressed learning via orthogonal matching pursuit with replacement (CL-OMPR) [5,6]. It aims to solve arg min…”

“…That is, {x t } T t=1 are assumed to be drawn i.i.d. from the GMM distribution (5). To recover the centroids C [c 1 , .…”

“…As proposed in [5], the row-norms {g m } from (8) were drawn i.i.d. from the distribution (20) with shape parameter σ 2 .…”

Instance Optimal Decoding and the Restricted Isometry Property

“…Note that this type of error is often considered when introducing quantization [7,19]. Ideally, one has η = 0, however in some cases it can be considerably simpler to prove that the LRIP holds with a non-zero η [20]. The reader would note that the classical RIP is often expressed with a constant α = (1 − t) −1 where t < 1 is a small as possible.…”

Quantized Compressive K-Means

“…Interestingly, κ also defines a Reproducing Kernel Hilbert Space in which two probability density functions (pdfs) can be compared with a Maximum Mean Discrepancy (MMD) metric [22][23][24][25][26]. Equipped with the MMD metric, the Generalized Method of Moments [27] in (3) is equivalent to an infinite-dimensional Compressed Sensing [11] problem, where the "sparse" pdf underlying the data (e.g., approximated by few Diracs) is reconstructed from a small number of compressive, random linear pdf measurements: the sketch [26]. The method to solve (3) is thus inspired by the OMP(R) CS recovery algorithm, i.e., Orthogonal Matching Pursuit (with Replacement) [28,29].…”