Mean Shift Spectral Clustering using Kernel Entropy Component Analysis

Abstract

Clustering is an unsupervised pattern recognition technique for finding natural groups in data, whether it is a grouping of web pages found by a search engine or segmenting satellite images into different types of ground cover. There exists a variety of different ways to perform clustering ranging from heuristics rules designed for a specific dataset to general procedures which can be applied to all datasets with varying degrees of success. The k-means algorithm is a well known example of the latter approach that can be expected to give good results when the data is easily separable.Another example of general clustering procedures are spectral clustering methods which involve a non-linear data transformation that allows them to handle complex cluster structures. They are considered to be the state of the art in the clustering literature, but are computationally demanding and unable to handle large datasets. To overcome the size limitations, this thesis uses a two-stage approach. The first stage can be seen as a preprocessing to reduce the size of the input for the spectral clustering in the second stage.The preprocessing is accomplished by using a simple clustering method on the dataset. These clusters represents a partitioning and is a more natural way of representing the dataset than randomly selecting a subset of samples. One can then adjust the number of partitions so spectral clustering methods can be used.The procedure is called Mean Shift Spectral Clustering (MSSC) as the mean shift clustering algorithm is used in the first stage. Each partition found by mean shift consists of data points that are close to the same mode in the estimated probability density function. Hence the partitions will be representative of the geometric structure of the dataset.This thesis realizes for the first time the idea of spectral clustering based on the partitions found by mean shift. The Kernel Entropy Component Analysis (KECA) spectral clustering method, a recent development in the field of spectral clustering, is used for this purpose and compared with the better known Kernel Principal Component Analysis (KPCA) method.A comprehensive collection of MATLAB functions has been developed to allow the testing of this procedure which is able to handle large and high dimensional datasets. It is able to cluster images directly with as many feature vectors as there are pixels. The experiments show how the clustering accuracy varies as a function of the primary parameters. This gives a good overall characterization of the method and what can be expected when used for unfamiliar datasets.The MSSC procedure is shown to provide good clustering results when following some basic principles for selecting parameters.ii

Robust clustering using a kNN mode seeking ensemble

“…In mean shift and mode seeking in general each data point is connected to a mode (local maximum) of the probability density function (pdf) and each mode represents a cluster. Successful applications of mean shift include Microsoft's Kinect® computer vision system [11], object motion tracking [12], initalization of spectral clustering algorithms [13,14] and change detection in satellite radar images [15]. In addition more recent applications have shown good results in visualizing functional connectivity in the brain [16], semi-supervised learning [17] and fault detection [18].…”

Consensus Clustering Using kNN Mode Seeking

“…In the original paper a heuristic approach was used as the final step [19]. Agersborg and Jenssen expanded the concepts and used true spectral clustering and proposed to use different choices of parameters in each step [1].…”

“…Particularly interesting developments in this line of research for the purpose of this paper, are recent attempts by Ozertem et al [19] and by Agersborg and Jenssen [1] to couple the mean shift algorithm with spectral clustering [17,15]. The idea is to merge together the modes found by mean shift by a spectral clustering algorithm based on a matrix encoding similarities between every pair of modes.…”

“…This entails simply counting for each k whether or not pairs of data points in the data set belong to the same basin of attraction (mode), for then to compute the average over all k. Based on the consensus matrix, a hierarchical clustering approach similar to that used in [9] and [11] is utilized in order to obtain the final clustering result. -The second approach we investigate, is based on for each k to compute an information theoretic divergence measure between pairs of modes resulting in a similarity matrix between modes, for then to average over all k. Then, a spectral clustering procedure is executed on this matrix, similar to [1].…”