Exploiting parallelism to support scalable hierarchical clustering

Abstract

A distributed memory parallel version of the group average hierarchical agglomerative clustering algorithm is proposed to enable scaling the document clustering problem to large collections. Using standard message passing operations reduces interprocess communication while maintaining efficient load balancing. In a series of experiments using a subset of a standard Text REtrieval Conference (TREC) test collection, our parallel hierarchical clustering algorithm is shown to be scalable in terms of processors efficiently used and the collection size. Results show that our algorithm performs close to the expected O(n 2 /p) time on p processors rather than the worst-case O(n 3 /p) time. Furthermore, the O(n 2 /p) memory complexity per node allows larger collections to be clustered as the number of nodes increases. While partitioning algorithms such as k-means are trivially parallelizable, our results confirm those of other studies which showed that hierarchical algorithms produce significantly tighter clusters in the document clustering task. Finally, we show how our parallel hierarchical agglomerative clustering algorithm can be used as the clustering subroutine for a parallel version of the buckshot algorithm to cluster the complete TREC collection at near theoretical runtime expectations. IntroductionDocument clustering has long been considered as a means to potentially improve both retrieval effectiveness and efficiency; however, the intensive computation necessary to cluster the entire collection makes its application to large datasets difficult. Accordingly, there is little work on effectively clustering entire large, standard-text collections and less with the intent of using these clusterings to aid retrieval. Rather, much work has focused on either performing simplified clustering algorithms or only using partial clusterings such as clustering only the results for a given query.Clustering algorithms generally consist of a trade-off between accuracy and speed. Hierarchical agglomerative clustering algorithms calculate a full document-to-document similarity matrix. Their clusterings are typically viewed as more accurate than other types of clusterings; however, the computational complexity required for the algorithm's quadratic behavior makes it unrealistic for large document collections. Other clustering algorithms such as the k-means and single pass algorithms iteratively partition the data into clusters. Although these partitioning algorithms run in linear time, the assignment of documents to moving centroids produces different clusterings with each run. Some algorithms combine the accuracy of hierarchical agglomerative algorithms with the speed of partitioning algorithms to get an algorithm that is fast with reasonable accuracy. One such algorithm is the buckshot algorithm, which uses a hierarchical agglomerative algorithm as a clustering subroutine.We propose a hierarchical agglomerative clustering algorithm designed for a distributed memory system in which we use the message passing model to facilitate in...

A Scalable Hierarchical Clustering Algorithm Using Spark

“…Several efforts were taken to parallelize hierarchical clustering algorithm, relying on the advance of modern computer architectures and large-scale systems. Different platforms, including multi-core [8], GPU [24], MPI [25] as well as recently popularized MapReduce framework [26], [27], have all seen its implementation.…”

A Novel Parallel Algorithm for Clustering Documents Based on the Hierarchical Agglomerative Approach

“…Because of its quadratic computational complexity, hierarchical clustering algorithms are unpractical for large document collections. A number of algorithms that exploit parallelism in hierarchical clustering algorithms have been introduced in literature [3,4,5,7,8,9,10]. A parallel hierarchical clustering algorithm is introduced in [3] which is used as the clustering subroutine for a parallel buckshot algorithm.…”

“…A number of algorithms that exploit parallelism in hierarchical clustering algorithms have been introduced in literature [3,4,5,7,8,9,10]. A parallel hierarchical clustering algorithm is introduced in [3] which is used as the clustering subroutine for a parallel buckshot algorithm. A distributed clustering technique (RACHET) is developed [9] in which hierarchical clustering algorithms are used to generate local dendrograms.…”

“…Computing similarity : The average cluster similarity is computed based on the cosine similarity measure; the most commonly used similarity measure in document clustering [3,11]. Cosine similarity is a normalized metric because its values fall in [0,1].…”

“…Iteratively, the closest pair of clusters are merged until k clusters are formed. A similarity matrix is used to store cluster-tocluster similarities [3]. The basic hierarchical agglomerative algorithm is as follows: Compute the similarity matrix Let each data point be a cluster Repeat Merge the two closest clusters Update the proximity matrix Until the desired number of clusters is reached Figure 1.…”