Sketched clustering via hybrid approximate message passing

Abstract

In sketched clustering, a dataset of T samples is first sketched down to a vector of modest size, from which the centroids are subsequently extracted. Advantages include i) reduced storage complexity and ii) centroid extraction complexity independent of T . For the sketching methodology recently proposed by Keriven et al., which can be interpreted as a random sampling of the empirical characteristic function, we propose a sketched clustering algorithm based on approximate message passing. Numerical experiments suggest that our approach is more efficient than the state-of-the-art sketched clustering algorithm "CL-OMPR" (in both computational and sample complexity) and more efficient than k-means++ when T is large.Index Terms-clustering algorithms, data compression, compressed sensing, approximate message passing * E. Byrne (byrne.133@osu.edu) and P. Schniter (schniter.1@osu.edu) are with the

Sketching Data Sets for Large-Scale Learning: Keeping only what you need

“…When n is very large, however, Lloyd's algorithm becomes computationally demanding. Instead, one could sketch the dataset using (1) and extract the centroids from the sketch [11,12]. For this purpose, one could use RF features (3) and (as explained in the sequel) solve for the centroids {c } k =1 and the (non-negative, sum-to-one) weights {α } k =1 that minimize z − k =1 α Φ(c ) .…”

“…In this setting, the weights allow us to model unbalanced clusters, yet only the centroids need to be recovered. On large datasets, this approach can be orders-of-magnitude better than Lloyd's algorithm in memory and runtime, provided that the sketch dimension m is large enough, i.e., that m is on the order of kd, where kd is the number of free parameters in {c } k =1 [11,12]. For example, this method allows us to cluster the MNIST digit dataset, of dimension d = 784 and cardinality n = 70 000, using a complex-valued sketch of dimension m = 400 (see Box 2).…”

“…These approaches sequentially estimate and subtract, from z, each of the k components α A(p θ ) that best align with it, where "best" is measured via the correlation between z and A(p θ ). As another example, an iterative approach [12] was proposed that exploits the log-likelihood interpretation of C(θ| z) in (18) and the i.i.d. random nature of the linear transform W in the RF map (3).…”

“…The dimension m of the sketch for which the LRIP holds with high probability is thus on the order of kd/c 0 , up to some additional factors due to the proof technique [5,6]. Empirical studies of compressive clustering [11,12] and compressive GMM [8] suggest that a sketch dimension m on the order of kd (the number of parameters in these settings) is sufficient to yield accurate learning performance. The best known bounds on provably good sketch dimensions [5,6] remain pessimistic compared to these empirically validated sketch dimensions.…”

“…When discussing methods to learn from a sketch, we focused on optimization-based approaches. Although heuristics based on orthogonal matching pursuit [8] and approximate message passing [12] have been proposed that yield promising empirical results, performance guarantees for these approaches have yet to be established. Alternatives, such as total-variation minimization over the space of signed measures [25,28,55], principled greedy methods [56], and gradient flows on systems of particles [57] could be leveraged to make progress on this front, and black-box optimization could be used to learn from sketches computed by optical processing units.…”