An efficient hybrid distributed document clustering algorithm

Abstract

Recent advances in information technology have led to an increase in volumes of data thereby exceeding beyond petabytes. Clustering distributed document sets from a central location is difficult due to the massive demand of computational resources. So there is a need for distributed document clustering algorithms to cluster documents using distributed resources. The greatest challenge in this area of distributed document clustering is the clustering quality and speedup associated with increase in document sets. The proposed clustering algorithm uses a hybrid algorithm which comprises of Particle Swarm Optimization (PSO), K-Means clustering and Latent Semantic Indexing (LSI) algorithm (PKMeansLSI), and uses MapReduce framework for distributed computation. The resultant of this is that it ultimately promotes clustering quality of the algorithm. The MapReduce framework and its corresponding implementation Hadoop is used as a distributed programming model which stresses on the improvement factor of the speedup of algorithm. The execution time is dramatically reduced as the dimensionality of documents is reduced. Experiment results show improved quality and effectiveness of the hybrid algorithm with varying increase in document size.

MapReduce-enhanced Fuzzy K-Least Medians for Qualitative Clustering of Document Big Data

“…The vector space model (VSM) is a widely practiced model for document preprocessing [22]. Particle Swarm Optimization (PSO) and Latent Semantic Indexing based optimization of MRK-Means are implemented in [23] on the Reuters-21578 dataset after the dataset is preprocessed using the VSM. The PSO is used for selecting the best centroids and dimensionality reduction.…”