Accurate Recasting of Parameter Estimation Algorithms Using Sufficient Statistics for Efficient Parallel Speed-Up

Abstract

Fueled by advances in computer technology and online business, data collection is rapidly accelerating, as well as the importance of its analysis-data mining. Increasing database sizes strain the scalability of many data mining algorithms. Data clustering is one of the fundamental techniques in data mining solutions. The many clustering algorithms developed face new challenges with growing data sets. Algorithms with quadratic or higher computational complexity, such as agglomerative algorithms, drop out quickly. More efficient algorithms, such as K-Means EM with linear cost per iteration, still need work to scale up to large data sets. This paper shows that many parameter estimation algorithms, including K-Means, K-Harmonic Means and EM, can be recast without approximation in terms of Sufficient Statistics, yielding an superior speed-up efficiency. Estimates using today's workstations and local area network technology suggest efficient speed-up to several hundred computers, leading to effective scale-up for clustering hundreds of gigabytes of data. Implementation of parallel clustering has been done in a parallel programming language, ZPL. Experimental results show above 90% utilization.

Distributed data clustering can be efficient and exact

“…In our previous paper [ZHF00], we described a parallel decomposition for center-based clustering algorithms that limits inter-processor communication to sufficient statistics only, reducing the network bottleneck. The data set is partitioned randomly across the memory of the processors and does not need to be transferred between iterations.…”

“…In a companion paper [ZHF00], we developed a class of parallel iterative parameter estimation algorithms, covering the centerbased clustering algorithms K-Means [M67] [GG92], KHarmonic Means [ZHD00a] [Z00b], and EM [DLR77] [MK97]. The parallelization is resource efficient and operates without approximation to the original sequential algorithms.…”

LBGS: a smart approach for very large data sets vector quantization

“…In this section, we will present a brief overview of some VQ techniques, both serial (LBG [29], Kmeans [31] and ELBG [36]) and parallel (PKM [38,39,1,9,43,11,34,33], PARELBG [34,37], P-CLUSTER [23][24][25] and PAUL [4]), selected from the large existing literature.…”

“…Various hardware architectures have been employed such as, for example: specialized architectures [33], massively parallel processors [38], transputers [39,1] and networks of workstations [9,43,11,34]. The idea at the basis of such techniques is the subdivision of the most timeconsuming part of the algorithm (the calculation of the Voronoi partition) into a certain number of subtasks to be executed in parallel, while, the remaining operations (the calculation of the new centroids) are serially executed by a single process.…”

“…When the task is very complex, i.e., both large data sets and large codebooks are involved (for example, in image segmentation [21], image coding and speech coding [42,30]), the computing time required by a classical approach [29,36] may be prohibitive. A possible solution to this problem is the use of parallel and distributed computing systems [38,39,1,9,43,11,24]. Recent studies [34,37] have shown that, for an efficient parallel implementation of the most advanced (serial) VQ techniques, it is necessary to use computing systems whose hardware provides a large bandwidth for inter-process communications.…”