Scalable Density-Based Distributed Clustering

Abstract

Abstract. Clustering has become an increasingly important task in analysing huge amounts of data. Traditional applications require that all data has to be located at the site where it is scrutinized. Nowadays, large amounts of heterogeneous, complex data reside on different, independently working computers which are connected to each other via local or wide area networks. In this paper, we propose a scalable density-based distributed clustering algorithm which allows a user-defined trade-off between clustering quality and the number of transmitted objects from the different local sites to a global server site. Our approach consists of the following steps: First, we order all objects located at a local site according to a quality criterion reflecting their suitability to serve as local representatives. Then we send the best of these representatives to a server site where they are clustered with a slightly enhanced density-based clustering algorithm. This approach is very efficient, because the local determination of suitable representatives can be carried out quickly and independently from each other. Furthermore, based on the scalable number of the most suitable local representatives, the global clustering can be done very effectively and efficiently. In our experimental evaluation, we will show that our new scalable density-based distributed clustering approach results in high quality clusterings with scalable transmission cost.

Using an MST based Value for ε in DBSCAN Algorithm for Obtaining Better Result

“…(1) mark all objects as unvisited; (2) do (3) randomly select an unvisited object p; (4) mark p as visited; (5 (6) create a new cluster C, and add p to C; (7) let N be the set of objects in the epsneighborhood of p; (8) for each point p' in N (9) if p' is unvisited (10) mark p' as visited; (11) if the eps-neighborhood of p' has at least MinPts points, add those points to N; (12) if p'…”

“…For instance in Fig.1,the points 1 , 2 , … … … … , 12 ,and q is not directly density-reachable by the point p. Centering at the point p, a circle defined by a given eps-neighborhood containing at least two other points The point 1 is directly densityreachable from p, and it is used as a center point to draw the next circle with the same eps-neighborhood. The same neighborhood-forming mechanism repeatedly applies to point p2 (which is directly density-reachable from p1) 3 , … … … … , 12 and q in sequence, as shown by the gray circles. However, in Web opinion clustering [30], it may create problems.…”

Scalable Clustering Algorithm for N-Body Simulations in a Shared-Nothing Cluster

“…FoF is a special case of the DBSCAN algorithm [23] corresponding to a M inP ts parameter of zero; there exists a large body of work on distributed DBSCAN algorithms [11,13,14,24]. This prior work can be categorized into two groups.…”

“…These algorithms build a distributed spatial index on a shared-nothing cluster and use this index when merging local clustering results [11,24]. The second category of approaches is to perform approximate clustering by using clustering on local models [13] or using samples to reduce the size of data or the number of spatial index lookups [14].…”

“…Additionally, previous work on parallelizing DBSCAN has been evaluated against relatively small and often synthetic datasets [11,13,14,24]. Their datasets have, at most, on the order of one million objects in two dimensions.…”