Fast clustering using MapReduce

Abstract

Clustering problems have numerous applications and are becoming more challenging as the size of the data increases. In this paper, we consider designing clustering algorithms that can be used in MapReduce, the most popular programming environment for processing large datasets. We focus on the practical and popular clustering problems, k-center and k-median. We develop fast clustering algorithms with constant factor approximation guarantees. From a theoretical perspective, we give the first analysis that shows several clustering algorithms are in MRC 0 , a theoretical MapReduce class introduced by Karloff et al. [26]. Our algorithms use sampling to decrease the data size and they run a time consuming clustering algorithm such as local search or Lloyd's algorithm on the resulting data set. Our algorithms have sufficient flexibility to be used in practice since they run in a constant number of MapReduce rounds. We complement these results by performing experiments using our algorithms. We compare the empirical performance of our algorithms to several sequential and parallel algorithms for the k-median problem. The experiments show that our algorithms' solutions are similar to or better than the other algorithms' solutions. Furthermore, on data sets that are sufficiently large, our algorithms are faster than the other parallel algorithms that we tested. which renders sequential algorithms unusable. In situations where the amount of data is prohibitively large, the MapReduce [16] programming paradigm is used to overcome this obstacle. MapReduce and its open source counterpart Hadoop [33] are distributed computing frameworks designed to process massive data sets.The MapReduce model is quite novel, since it interleaves sequential and parallel computation. Succinctly, MapReduce consists of several rounds of computation. There is a set of machines, each of which has a certain amount of memory available. The memory on each machine is limited, and there is no communication between the machines during a round. In each round, the data is distributed among the machines. The data assigned to a single machine is constrained to be sub-linear in the input size. This restriction is motivated by the fact that the input size is assumed to be very large [26,15]. After the data is distributed, each of the machines performs some computation on the data that is available to them. The output of these computations is either the final result or it becomes the input of another MapReduce round. A more precise overview of the MapReduce model is given in Section 1.1. Problems:In this paper, we are concerned with designing clustering algorithms that can be implemented using MapReduce. In particular, we focus on two well-studied problems: metric k-median and k-center. In both of these problems, we are given a set V of n points, together with the distances between any pair of points; we give a precise description of the input representation below. The goal is to choose k of the points. Each of the k chosen points represents a cluster and is refer...

Big Data Analysis for Media Production

“…Distributed versions of clustering algorithms related to kernel k-Means, like classic k-Means [86] and k-Medians [87] have already been proposed. However, to the best of our knowledge, a distributed approach to kernel k-Means has not been proposed yet.…”

Summarizing large text collection using topic modeling and clustering based on MapReduce framework

“…MapReduce framework is widely used for processing and managing large data sets in a distributed cluster, which has been used for numerous applications such as, document clustering, access log analysis, generating search indexes and various other data analytical operations. A host of literature is present in recent years for performing Big Data clustering using MapReduce framework [3,4,[13][14][15][16]. A modified K-means clustering algorithm based on MapReduce framework is proposed by Li et al [17] to perform clustering on large data sets.…”

Efficient <inline-formula><tex-math>$k$</tex-math><alternatives> <inline-graphic xlink:type="simple" xlink:href="qu-ieq1-2306193.gif"/></alternatives></inline-formula>-Means++ Approximation with MapReduce

“…Papadimitriou et al presented the distributed co-clustering framework which introduced practical approaches for distributed data preprocessing and co-clustering [11]. Ene et al proposed the fast clustering using MapReduce [12] by adopting a MapReduce sampling technique to decrease the data size. The result of this method was applied to -center andmedian algorithm.…”

“…We have discussed above, this algorithm needs too many MapReduce jobs. The research in [12] proposed a fast clustering scheme which uses the sapling technology. This paper also proved that the MapReduce-KCenter was 4 + 2 for the -center problem and MapReduce-KMedian was 10 + 3 approximation for the -median problem.…”