Parallel Density-Based Clustering of Complex Objects

Abstract

Abstract. In many scientific, engineering or multimedia applications, complex distance functions are used to measure similarity accurately. Furthermore, there often exist simpler lower-bounding distance functions, which can be computed much more efficiently. In this paper, we will show how these simple distance functions can be used to parallelize the density-based clustering algorithm DBSCAN. First, the data is partitioned based on an enumeration calculated by the hierarchical clustering algorithm OPTICS, so that similar objects have adjacent enumeration values. We use the fact that clustering based on lower-bounding distance values conservatively approximates the exact clustering. By integrating the multi-step query processing paradigm directly into the clustering algorithms, the clustering on the slaves can be carried out very efficiently. Finally, we show that the different result sets computed by the various slaves can effectively and efficiently be merged to a global result by means of cluster connectivity graphs. In an experimental evaluation based on real-world test data sets, we demonstrate the benefits of our approach.

Cludoop: An Efficient Distributed Density-Based Clustering for Big Data Using Hadoop

“…The experiment shows the algorithm achieves near linear speedup. Brecheisen et al [10] present a parallel DBSCAN on a workstation, which is parallelized by a conservative approximation of complex distance functions, based on the concept of filter merge points. The final result is derived from a global cluster connectivity graph.…”

“…= Unclassified then (7) if . = Unclassified then (8) Create c-cluster ← ( ); (9) if expandcluster ( , , , , , ) then (10) Create c-cluster ← ( ++); (11) end if (12) end if (13) the cells according to the received points and given in each of the mappers, so the cell with the same in different mappers stands for the same area; thus we can only use the cell to locate the assigned range in overall data space. (2) The points in an inclusive cell must belong to a c-cluster, so the cell and c-cluster are enough to stand for classification of all points in the cell.…”

A new scalable parallel DBSCAN algorithm using the disjoint-set data structure

“…The algorithm starts with an arbitrary point x ∈ X and retrieves its eps-neighborhood (Line 4). If the epsneighborhood contains at least minpts points, the procedure yields a new cluster, C. The algorithm then retrieves all points in X, which are density reachable from x and adds them to the cluster C (Line [8][9][10][11][12][13][14][15][16][17]. If the eps-neighborhood of x has less than minpts, then x is marked as noise (Line 6).…”

“…The speedups are plotted in Figure 5 Finally, we have compared our parallel DBSCAN algorithm with the previous master-slave approaches [14], [15], [16], [17], [18], [19], [20]. As their source codes are not available, we have implemented their ideas, where the master process perform the cluster assignment while the slave processes answer the neighborhood queries [15], [17].…”

“…Many existing parallelizations adopt the master-slave model. For example, in [14], the data is equally partitioned and distributed among the slaves, each of which computes the clusters locally and sends back the results to the master in which the partially clustered results are merged sequentially to obtain the final result. This strategy incurs a high communication overhead between the master and slaves, and a low parallel efficiency during the merging process.…”