Feature selection for clustering categorical data with an embedded modelling approach

Abstract

Research on the problem of feature selection for clustering continues to develop. This is a challenging task, mainly due to the absence of class labels to guide the search for relevant features. Categorical feature selection for clustering has rarely been addressed in the literature, with most of the proposed approaches having focused on numerical data. In this work, we propose an approach to simultaneously cluster categorical data and select a subset of relevant features. Our approach is based on a modification of a finite mixture model (of multinomial distributions), where a set of latent variables indicate the relevance of each feature. To estimate the model parameters, we implement a variant of the expectation-maximization algorithm that simultaneously selects the subset of relevant features, using a minimum message length criterion. The proposed approach compares favourably with two baseline methods: a filter based on an entropy measure and a wrapper based on mutual information. The results obtained on synthetic data illustrate the ability of the proposed expectation-maximization method to recover ground truth. An application to real data, referred to official statistics, shows its usefulness.

Efficient mixture model for clustering of sparse high dimensional binary data

“…Coding criteria usually are used for comparing two models (like AIC or BIC criteria). Silvestre et al (2015) showed how to apply the MML criterion simultaneously with a clustering method. This is similar to our algorithm, which reduces redundant clusters on-line.…”

Detecting brain network communities: considering the role of information flow and its different temporal scales

“…On the other hand, the selection of features that minimize redundancy is superior to feature reduction in terms of interpretability (Alelyani Salem et al, 2013) and performance (Ronan et al, 2016). Problems like the one our method is concerned with, binary feature selection for clustering, have rarely been addressed though, while most of the studies have focussed on numerical variables (Silvestre, Cardoso, & Figueiredo, 2015). To our knowledge, only a handful of works did explore clustering in the presence of categorical (thus also binary, in particular) data (Bontemps & Toussile, 2013;Silvestre et al, 2015).…”

“…Problems like the one our method is concerned with, binary feature selection for clustering, have rarely been addressed though, while most of the studies have focussed on numerical variables (Silvestre, Cardoso, & Figueiredo, 2015). To our knowledge, only a handful of works did explore clustering in the presence of categorical (thus also binary, in particular) data (Bontemps & Toussile, 2013;Silvestre et al, 2015). The methods therein developed make certain assumptions on the data and only solve the feature selection problem by simultaneously targeting a distribution in the desired number of clusters, which would not straightforwardly align with the rest of the pipeline in our algorithm.…”

Recent advances on knowledge discovery and business intelligence

“…In 'Feature selection for clustering categorical data with an embedded modelling approach', Silvestre et al (2014) present a novel approach that simultaneously clusters categorical data and selects relevant features. The approach is based on a Gaussian mixture model, where the minimum message length criterion is used to guide the selection of the relevant features and a modified expectation-maximization algorithm estimates the model parameters.…”