Hierarchical document clustering using local patterns

Abstract

The global pattern mining step in existing pattern-based hierarchical clustering algorithms may result in an unpredictable number of patterns. In this paper, we propose IDHC, a pattern-based hierarchical clustering algorithm that builds a cluster hierarchy without mining for globally significant patterns. IDHC first discovers locally promising patterns by allowing each instance to "vote" for its representative size-2 patterns in a way that ensures an effective balance between local pattern frequency and pattern significance in the dataset. The cluster hierarchy (i.e., the global model) is then directly constructed using these locally promising patterns as features. Each pattern forms an initial (possibly overlapping) cluster, and the rest of the cluster hierarchy is obtained by following a unique iterative cluster refinement process. By effectively utilizing instance-to-cluster relationships, this process directly identifies clusters for each level in the hierarchy, and efficiently prunes duplicate clusters. Furthermore, IDHC produces cluster labels that are more descriptive (patterns are not artificially Responsible editor: Johannes Fürnkranz and Arno Knobbe. 123 154 H. H. Malik et al.restricted), and adapts a soft clustering scheme that allows instances to exist in suitable nodes at various levels in the cluster hierarchy. We present results of experiments performed on 16 standard text datasets, and show that IDHC outperforms state-of-the-art hierarchical clustering algorithms in terms of average entropy and FScore measures.

Parallel N-Path Quantification Hierarchical K-Means Clustering Algorithm for Video Retrieval

“…Malik et al proposed a pattern-based instance-driven hierarchical clustering algorithm (IDHC) that builds a cluster hierarchy without mining for globally signi¯cant patterns. 21 While Antonio et al presented a divisive hierarchical method which is based on the use of the k-means method embedded in a recursive algorithm to obtain a clustering at each node of the hierarchy. 4 Liao et al introduces a sample-based hierarchical adaptive K-means (SHAKM) clustering algorithm, which employs multilevel random sampling to handle large databases and utilizes the adaptive K-means clustering algorithm to determine the correct number of clusters.…”

Bringing Order to Legal Documents - An Issue-based Recommendation System Via Cluster Association

“…More recently Fukumoto and Suzuki performed cluster labeling by relying on concepts in a machine readable dictionary (Fukumoto and Suzuki, 2011) with positive results. In another distinct recent work, Malik, et al, focused on finding patterns (i.e., labels) and clusters simultaneously as an alternative to explicitly identifying labels for existing clusters (Malik et al, 2010).…”

Automatic Training Data Cleaning for Text Classification

“…The number of clusters k was set to obtain clusters that averaged 100 documents (i.e., if clustering 1000 documents, k = 10). We additionally used a more recent clustering algorithm, IDHC [9], which is different than k-means in that it does not take a parameter k and produces a variable number of clusters.…”