Kernel Clustering: Density Biases and Solutions

Abstract

Abstract-Kernel methods are popular in clustering due to their generality and discriminating power. However, we show that many kernel clustering criteria have density biases theoretically explaining some practically significant artifacts empirically observed in the past. For example, we provide conditions and formally prove the density mode isolation bias in kernel K-means for a common class of kernels. We call it Breiman's bias due to its similarity to the histogram mode isolation previously discovered by Breiman in decision tree learning with Gini impurity. We also extend our analysis to other popular kernel clustering methods, e.g. average/normalized cut or dominant sets, where density biases can take different forms. For example, splitting isolated points by cut-based criteria is essentially the sparsest subset bias, which is the opposite of the density mode bias. Our findings suggest that a principled solution for density biases in kernel clustering should directly address data inhomogeneity. We show that density equalization can be implicitly achieved using either locally adaptive weights or locally adaptive kernels. Moreover, density equalization makes many popular kernel clustering objectives equivalent. Our synthetic and real data experiments illustrate density biases and proposed solutions. We anticipate that theoretical understanding of kernel clustering limitations and their principled solutions will be important for a broad spectrum of data analysis applications across the disciplines.

Deep Clustering: On the Link Between Discriminative Models and K-Means

“…We recall here that the equivalence was proved assuming the clusters are balanced. Also, notice that MI-ADM and DEPICT have approximately the same performance, confirming our earlier discussion: MI-ADM in (13) can be viewed as an approximation of DEPICT in (6). The additional entropy term in (13), H(Q), has almost no effect of the results.…”

“…Also, notice that MI-ADM and DEPICT have approximately the same performance, confirming our earlier discussion: MI-ADM in (13) can be viewed as an approximation of DEPICT in (6). The additional entropy term in (13), H(Q), has almost no effect of the results. Finally, notice the substantial difference in performance (11%) between our regularized and soft Kmeans and DCN [3], which is based on a hard K-means loss.…”

“…NMI translates the similarity between pairs of clusters, and is invariant w.r.t permutations [26]. Table 1 reports the results 5 of discriminative model MI-ADM in (13) and generative model SR-K-means in (18). We also include the results of several related models: (1) the DEPICT model [4] based on KL and logistic regression posteriors, which achieves a state-of-the-art performance on MNIST;…”

Iterative Min Cut Clustering Based on Graph Cuts

“…Kernel method maps a nonlinearly separable dataset into a higher-dimensional Hilbert space, and in the Hilbert space the dataset may be linearly separable. DBK clustering [15] proposes a density equalization principle, and then based on this principle, they propose an adaptive kernel clustering algorithm. Multiple kernels clustering algorithms [16][17][18][19] use multiple kernel functions to enhance the performance of kernel clustering algorithms.…”

An Internal Validity Index Based on Density-Involved Distance

“…In the experiments, we discard some extreme partitions from M partitions with Ncut's mode isolation (Breiman's bias[76]), namely, partitions possess clusters that are composed of few objects.…”