Simultaneous Clustering and Model Selection for Multinomial Distribution: A Comparative Study

Abstract

Abstract. In this paper, we study different discrete data clustering methods, which use the Model-Based Clustering (MBC) framework with the Multinomial distribution. Our study comprises several relevant issues, such as initialization, model estimation and model selection. Additionally, we propose a novel MBC method by efficiently combining the partitional and hierarchical clustering techniques. We conduct experiments on both synthetic and real data and evaluate the methods using accuracy, stability and computation time. Our study identifies appropriate strategies to be used for discrete data analysis with the MBC methods. Moreover, our proposed method is very competitive w.r.t. clustering accuracy and better w.r.t. stability and computation time.

Evolutionary clustering for categorical data using parametric links among multinomial mixture models

“…The Multinomial Mixture (MM) is a statistical model based on the Multinomial distribution. It has been used for cluster analysis with discrete data (Meilȃ and Heckerman, 2001;Zhong and Ghosh, 2005;Hasnat et al, 2015). Meilȃ and Heckerman (2001) studied several Model-Based Clustering (MBC) methods with the MM and experimentally compared them using different criteria such as clustering accuracy, computation time and number of selected clusters.…”

“…Silvestre et al (2014) proposed a MBC method, which integrates both model estimation and selection within a single EM algorithm. Recently, Hasnat et al (2015) proposed a MBC method which performs simultaneous clustering and model selection using the MM. Their strategy performs similar task as Silvestre et al (2014) in a computationally efficient manner, which has been previously proposed for the Gaussian distribution (Garcia and Nielsen, 2010) and the Fisher distribution (Hasnat et al, 2016).…”

“…With different initializations the EM algorithm may converge to different values of the likelihood function, some of which can be local maxima (i.e., sub-optimal results). In order to overcome this, numerous different initialization strategies are proposed and experimented in the relevant literature (Biernacki et al, 2003;Meilȃ and Heckerman, 2001;Baudry and Celeux, 2015;Hasnat et al, 2015). Following recommendations, we use the small-EM (Biernacki et al, 2003(Biernacki et al, , 2006Baudry and Celeux, 2015;Hasnat et al, 2015) method to initialize the MM parameters.…”

“…Generally, the MM parameters are initialized randomly (Meilȃ and Heckerman, 2001;Hasnat et al, 2015). However, with both synthetic and real data it has been demonstrated by Hasnat et al (2015) that, random initialization has its limitation w.r.t. the clustering performance and stability.…”

“…the clustering performance and stability. Therefore, following Hasnat et al (2015), we initialize the model parameters using the small-EM procedure. It consists of running multiple short runs of randomly initialized EM and then selecting the one with the maximum-likelihood value.…”