A new approach to clustering based on statistical physics is presented. The problem is formulated as fuzzy clustering and the association probability distribution is obtained by maximizing the entropy at a given average variance. The corresponding Lagrange multiplier is related to the "temperature" and motivates a deterministic annealing process where the free energy is minimized at each temperature. Critical temperatures are derived for phase transitions when existing clusters split. It is a hierarchical clustering estimating the most probable cluster parameters at various average variances.

No abstract

No abstract

at Santa Barbara, as respectively an assistant professor. His research interests are in pattern recognition, Since 1967, he has been at the Nuclear Research information theory, neural networks, image coding and processing, and Center-Negev, Beer Sheva, Israel, doing research nonconvex optimization in general. on neutron scattering, magnetic structures group Dr. Rose received the 1990 William R. Bennett Prize Paper Award of theory, and phase transitions. During the past five the IEEE Communications Society for his paper "Enhancement of One-years, his research has been concentrated in the Dimensional Variable-Length DPCM Images Corrupted by Transmission areas of computational physics, optimization, parallel processing, and image Errors" (with Arie Heiman) in the April 1989 IEEE TRANSACTIONS ON processing COMMUNICATIONS. Dr Gurewitz is a member of the Israel Physical Society.

J. IntroductionClustering is usually formulated as an optimization problem by defining a cost function to be minimized. Most of these cost functions are nonconvex and have several local minima Traditional techniques [1,2,3,4] are essentially decent algorithms, as the cost is reduced at each iteration. Therefore, they tend to get trapped in local minima.Simulated annealing, or stochastic relaxation [5], is a known technique for avoiding local minima of nonconvex optimization problems. However, the process to escape local minima requires very slow schedules [6] which are not realistic for many practical applications.In a previous work [7,8] we proposed the concept of deterministic annealing for the problem of clustering and vector quantization. Our approach is strongly motivated by the physical analogy and baeed on principals of statistical mechanics or information theory [9,10]. In this paper we extend our clustering method to the constraint clustering method. Sections 2 and 3 are a brief presentation of our clustering approach. Section 4 presents the "constraint clustering approach", and sections 5 and 6 are two examples for which this approach can be applied. Clustering by Deterministic AnnealingIn previous works [7,8] we have suggested a deterministic annealing approach to clustering and vector quantization. This approach is briefly summarized in this section and the following one.The method uses a fuzzy formulation where each data point z is associated in probability with each cluster Cj. The cluster C, is represented by its "cluster centroid" yj. The energy (cost or distortion) of associating the data point z to the cluster C, is d ( z , y , ) . If the set Y = {yj} of the clusters representatives is given, the expected energy of the system is where P(z E Cj) is the probability that the data point z belongs to the cluster Cj. Since we do nof have any prior knowledge about the data probability distribution function, we apply the principle of maximum entropy.As is well known, the probability distribution which maximizes the entropy under the expectation constraint (1) is e-Dd(z,Vj)where Zx is the partition function Z, = cj e-pd(zJ'J).The Lagrange multiplier p is determined by a given value of E in (l), and is inversely proportional to the "temperature". For a given set of cluster representatives {y,}, it is assumed that the probabilities relating different data point to their clusters are independent. Hence the total partition function is Z = n, Z,, and the free energy F is given by F = --logz(Y) = -C l o g Ce-@d(Z,Yj)