Compressive K-means

Abstract

The Lloyd-Max algorithm is a classical approach to perform Kmeans clustering. Unfortunately, its cost becomes prohibitive as the training dataset grows large. We propose a compressive version of K-means (CKM), that estimates cluster centers from a sketch, i.e. from a drastically compressed representation of the training dataset.We demonstrate empirically that CKM performs similarly to Lloyd-Max, for a sketch size proportional to the number of centroids times the ambient dimension, and independent of the size of the original dataset. Given the sketch, the computational complexity of CKM is also independent of the size of the dataset. Unlike Lloyd-Max which requires several replicates, we further demonstrate that CKM is almost insensitive to initialization. For a large dataset of 10 7 data points, we show that CKM can run two orders of magnitude faster than five replicates of Lloyd-Max, with similar clustering performance on artificial data. Finally, CKM achieves lower classification errors on handwritten digits classification.

Sketched clustering via hybrid approximate message passing

“…To recover the centroids C from y, the state-of-the-art algorithm is compressed learning via orthogonal matching pursuit with replacement (CL-OMPR) [5,6]. It aims to solve arg min…”

The basins of attraction of the global minimizers of the non-convex sparse spike estimation problem

“…Estimating such signals from a finite number of Fourier measurements is known as the super-resolution problem [9]. Also, the estimation of spikes from random Fourier measurements is at the core of the compressive K-means algorithm were k-means cluster centers are estimated from a compressed database [21]. In the space M of finite signed measure over R d , we aim at recovering x 0 = i=1,k a i δ t i from the measurements…”

Quantized Compressive K-Means

“…The sketch is thus the pooling (average) of random projections of the data samples after passing through a nonlinear, periodic signature-the complex exponential. The Compressive K-Means (CKM) method [8] clusters X from z X by replacing (1) with a sketch matching optimization problem:…”

“…For the related kernel K-means problem, [20] uses Random Fourier Features [21], i.e., the low-dimensional mapping z (·) defined in (2). For u, v ∈ R n , the inner product z u , z v approximates a shift-invariant kernel κ(u, v) associated with the frequency distribution Λ. CKM [8] actually averages individual RFF of data points. Interestingly, κ also defines a Reproducing Kernel Hilbert Space in which two probability density functions (pdfs) can be compared with a Maximum Mean Discrepancy (MMD) metric [22][23][24][25][26].…”

“…as announced in (3). This non-convex problem is hard to solve exactly, but the CKM algorithm (based on OMPR) [8], detailed in pseudocode below, seeks an approximate solution. More precisely, CKM greedily selects new centroids minimizing a residual r ∈ C m (Steps 1 and 2) inside a box with lower and upper bounds l, u ∈ R n , respectively, enclosing the data X, and eventually replacing bad centroids in Step 3.…”