Enhanced word clustering for hierarchical text classification

Abstract

In this paper we propose a new information-theoretic divisive algorithm for word clustering applied to text classification. In previous work, such "distributional clustering" of features has been found to achieve improvements over feature selection in terms of classification accuracy, especially at lower number of features [2,28]. However the existing clustering, techniques are agglomerative in nature and result in (i) sub-optimal word clusters and (ii) high computational cost. In order to explicitly capture the optimality of word clusters in an information theoretic framework, we first derive a global criterion for feature clustering. We then present a fast, divisive algorithm that monotonically decreases this objective function value, thus converging to a local minimum. We show that our algorithm minimizes the "within-cluster Jensen-Shannon divergence" while simultaneously maximizing the "between-cluster Jensen-Shannon divergence". In comparison to the previously proposed agglomerative strategies our divisive algorithm achieves higher classification accuracy especially at lower number of features. We further show that feature clustering is an effective technique for building smaller class models in hierarchical classification. We present detailed experimental results using Naive Bayes and Support Vector Machines on the 20 Newsgroups data set and a 3-level hierarchy of HTML documents collected from Dmoz Open Directory.

Text Classification Aided by Clustering: a Literature Review

“…The works of (Slonim & Tishby, 2000), , (Yaniv & Souroujon, 2001) use heuristic procedures to cluster documents and features independently using an agglomerative algorithm. (Dhillon et al, 2002(Dhillon et al, , 2003b on the other hand, propose an information-theoretic coclustering algorithm that intertwines both row (feature) and column (document) clustering. The algorithm starts with a random partition of rows, X, and columns, Y, and computes an approximation q(X,Y) to the original distribution P(X,Y) and a corresponding compressed distribution by co-clustering rows and columns intertwined, i.e.…”

“…Experiments conducted demonstrate the efficiency of the algorithm especially in the presence of sparsity. (Dai et al, 2007) extend the co-clustering algorithm of (Dhillon et al, 2002(Dhillon et al, , 2003b and present a co-clustering classification algorithm (CoCC) that focuses on classifying documents across different text domains. There is a labelled data set D i from one domain, called in-domain, and an unlabelled data set D o from a related but different domain, called out-of-domain, that is to be classified.…”

Semi-supervised model-based document clustering: A comparative study

“…A connection between multinomial model-based clustering and the divisive KullbackLeibler clustering (Dhillon et al, 2002;Dhillon & Guan, 2003) is worth mentioning here. It is briefly mentioned in Dhillon and Guan (2003) but they did not explicitly stress that the divisive KL clustering is equivalent to multinomial model-based k-means, which maximizes the following objective function:…”

Web Page Classification*

“…24;Rasmussen 1992;Silverstein and Pedersen 1997;Jain et al 1999;Manning and Schutze 2001: ch. 14;Everitt et al 2001;Duda et al 2001;Dillon et al 2002;Berkhin 2000;Yao and Choi 2003).…”