Cluster Identification Using Projections

Abstract

This artiele describes a procedure to identify elusters in multivariate data using information obtained from the univariate proj~ctions of the sample data onto certain directions. The directions are chosen as those that minimize and maXlmlze the kurtOSlS coefficlent of the projected data. It is shown that, under certain conditions, these directions provide the largest separatlOn for the dlfferent clusters. The projected univariate data are used to group the observations according to the values of the gaps or spacmgs between consecutIveordered observations. These groupings are then combined over all projection directions. The behavlOr of the method lS tested on ~everal examples, and compared to k-means, MCLUST, and the procedure proposed by Jones and Sibson in 1987. The proposed algonthm lS iterative, affine equivariant, flexible, robust to outliers, fast to implement, and seems to work well m practIce.KEY WORDS: Classification; Kurtosis; Multivariate analysis; Robustness; Spacings. . INTRODUCTIONLet us suppose we have a sample of multivariate observations generated from several different populations. One of the most important problems of cluster analysis is the partitioning of the points of this sample into nonoverlapping clusters. The most commonly used algorithms as sume that the number of clusters, G, is known and the partition of the data is carried out by maximizing some optimality criterion. These algorithms start with an initial classification of the points into clusters and then reassign each point in tum to increase the criterion. The process is repeated until a local optimum of the criterion is reached. The most often used criteria can be derived from the application of likelihood ratio tests to mixtures of multivariate normal populations with different means. It is well known that (i) when all the covariance matrices are assumed to be egual to the identity matrix, the criterio n obtained corresponds to minimizing tr(W), where W is the within-groups covariance matrix, this is the criterion used in the standard k-means procedure; (ii) when the covariance matrices are assumed to be egual, without other restrictions, the criterion obtained is minimizing ¡W¡ (Friedman and Rubin 1967); (iii) when the covariance matrices are allowed to be different, the criterion obtained is minimizing Lf= 1 n j log I W j / n J where W j is the sample cross-product matrix for the jth cluster (see Seber 1984, and Gordon 1994, for other criteria). These algorithms may present two main limitations: (i) we have to choose the criterion a priori, without knowing the covariance structure of the data and different criteria can lead to very different answers; and (ii) they usually reguire large amounts of computer time, which makes them difficult to apply to large data sets.Banfield and Raftery (1993), and Dasgupta and Raftery (1998) have proposed a model-based approach to clustering that has several advantages over previous procedures. They as sume a mixture model and use the EM algorithm to estimate the parameters. The initial esti...

Overview of object oriented data analysis

“…This elegant idea, first proposed by Cardoso and Soumouliac (1993), and described in detail in Oja et al (2001), is to replace the maximal variation directions of PCA with maximally non-Gaussian directions. See Peña and Prieto (2001) for a related idea. Such directions can be very useful for data visualization because they can focus on clusters, and other interesting aspects of the dataset.…”

A Projection Pursuit framework for supervised dimension reduction of high dimensional small sample datasets

“…A great deal of research in the PP community has been centered on the construction of meaningful PP indexes for different purposes. It is possible to find PP indexes for clustering analysis [42,19,38,39,43,30,13], for supervised analysis [44,45,21,5] and for regression analysis [46,47]. Given that this paper is targeted to supervised analysis, we briefly describe some relevant supervised PP indexes included in the experimental evaluation of the paper.…”

“…A key advantage of PP is its flexibility to fit different pattern recognition tasks, depending on the PP index used. For example, PP can be used to perform clustering analysis [19,20], classification [21][22][23][24], regression analysis [25] and density estimation [26] (some reviews of PP indexes can be found in [21,27,28]). Another advantage of PP is its out-of-sample mapping capability, that is, the possibility to map new examples in the projection space after the construction of it.…”

“…Evidently, the problem worsens as p or m increase. Traditional PP optimizers based on the gradients or Newton methods [29][30][31]19] are usually inadequate for such a kind of data due to the vastness of possible projections and, thus, the high susceptibility to find poor local optima [14]. More global PP optimizers were described recently, including genetic algorithms (GA) [32,33], simulating annealing (SA) [21], random scan sampling (RSSA) [34] and particle swarm optimization (PSO) [35].…”

Selection of Variables for Cluster Analysis and Classification Rules

“…In cluster analysis (or un-supervised classification) we look for a partition of the space into homogeneous groups or clusters (with small dispersion within groups), that help us to understand the structure of the data. Several cluster methods have been proposed, such as hierarchical clustering (Hartigan, 1975), k-means (MacQueen, 1967, kmediods (Kaufman and Rousseeuw, 1987), kurtosis based clustering (Peña and Prieto, 2001). From most of them we get a partition of the space in disjoint subsets.…”