MR-DBSCAN: a scalable MapReduce-based DBSCAN algorithm for heavily skewed data

He, Yaobin; Tan, Haoyu; Luo, W.; Feng, Shijie; Fan, Jianping

doi:10.1007/s11704-013-3158-3

Cited by 144 publications

(75 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2017, 9, 1301 5 of 32 (2) Density: at the position of data point p in the dataset D , the number of points, Num, contained in the neighborhood Eps is its density. , where there is a core point and this is inside the Eps neighborhood of q , it is said that from p to q is direct density-reachable, i.e., (7) Density-reachable: given objects 1 Figure 3.…”

Section: Mathematical Principles Of the Dbscan Algorithmmentioning

confidence: 99%

“…Remote Sens. 2017, 9, 1301 5 of 32 (2) Density: at the position of data point p in the dataset D , the number of points, Num, contained in the neighborhood Eps is its density. Figure 4.…”

Section: Mathematical Principles Of the Dbscan Algorithmmentioning

confidence: 99%

“…If q does not belong to any cluster, then add q to C 1 . (3) Repeat step (2), and continue to check object set N 1 until it is empty. (4) Repeat steps (1) to (3).…”

Section: Processing Procedures Of the Dbscan Algorithmmentioning

confidence: 99%

“…It has been widely used in the field of data mining. However, in some large-scale spatial data mining studies [2], as the scale of data computation increases, the processing time of DBSCAN rises exponentially. Thus, the performance of this serial algorithm is unable to meet the needs of real-time application development.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Research on the Parallelization of the DBSCAN Clustering Algorithm for Spatial Data Mining Based on the Spark Platform

et al. 2017

View full text Add to dashboard Cite

Density-based spatial clustering of applications with noise (DBSCAN) is a density-based clustering algorithm that has the characteristics of being able to discover clusters of any shape, effectively distinguishing noise points and naturally supporting spatial databases. DBSCAN has been widely used in the field of spatial data mining. This paper studies the parallelization design and realization of the DBSCAN algorithm based on the Spark platform, and solves the following problems that arise when computing macro data: the requirement of a great deal of calculation using the single-node algorithm; the low level of resource-utilization with the multi-node algorithm; the large time consumption; and the lack of instantaneity. The experimental results indicate that the proposed parallel algorithm design is able to achieve more stable speedup at an increased involved spatial data scale.

show abstract

Section: Mathematical Principles Of the Dbscan Algorithmmentioning

confidence: 99%

“…Remote Sens. 2017, 9, 1301 5 of 32 (2) Density: at the position of data point p in the dataset D , the number of points, Num, contained in the neighborhood Eps is its density. Figure 4.…”

Section: Mathematical Principles Of the Dbscan Algorithmmentioning

confidence: 99%

“…If q does not belong to any cluster, then add q to C 1 . (3) Repeat step (2), and continue to check object set N 1 until it is empty. (4) Repeat steps (1) to (3).…”

Section: Processing Procedures Of the Dbscan Algorithmmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Research on the Parallelization of the DBSCAN Clustering Algorithm for Spatial Data Mining Based on the Spark Platform

et al. 2017

View full text Add to dashboard Cite

show abstract

“…In addition, a core point reduction approach is proposed and implemented for more efficient and yet accurate data summarization and regeneration. Note that MapReduce-based implementation of DBSCAN algorithm has been studied in [8,9,10] that are relying on different partitioning strategies. In this thesis, a grid-based partitioning approach is used for an implementation of DBSCAN algorithm that is compatible with GMM summarization, which are both completely developed in the MapReduce framework.…”

Section: Contributionsmentioning

confidence: 99%

Distributed Gaussian Mixture Model Summarization Using the MapReduce Framework

Esmaeilpour

Bigdeli

Cheraghchi

et al. 2016

Advances in Artificial Intelligence

View full text Add to dashboard Cite

With an accelerating rate of data generation, sophisticated techniques are essential to meet scalability requirements. One of the promising avenues for handling large datasets is distributed storage and processing. Hadoop is a well-known framework for distributed storage and processing. Further, data summarization is a useful concept for managing large datasets. Data summarization techniques are intended to produce compact yet representative summaries for the entire dataset.Consolidation of these tools can allow a distributed implementation of data summarization. In this thesis, this goal is achieved by proposing and implementing a distributed Gaussian Mixture Model Summarization using the MapReduce framework (MR-SGMM). The main purpose of the proposed method is to summarize a dataset with a density-based clustering algorithm called DBSCAN algorithm, and then summarize each discovered cluster using the SGMM approach in a distributed manner. Testing the implementation with synthetic and real datasets is used to demonstrate its validity and efficiency.ii

show abstract

Big Data Clustering

Khondoker

2018

Wiley StatsRef: Statistics Reference Online

View full text Add to dashboard Cite

Clustering algorithms group data items based on clearly defined similarity between the items aiming to minimize the intracluster differences and maximize the intercluster distances. A wealth of efficient and good quality clustering algorithms are already available for traditional data, but there are challenges for applying them to big data due to the overwhelming volume and complexities of such data. Data volume is getting bigger at an incredible pace due to growing access to Internet, social media, mobile devices, and technological innovations, and improving clustering algorithms, their computational cost and scalability have been the focus of much of the research in this area. This article provides an introduction to the characteristics of big data, and an overview of available algorithms and the current improvement trend of clustering algorithms for dealing with the challenges of big data.

show abstract

MR-DBSCAN: a scalable MapReduce-based DBSCAN algorithm for heavily skewed data

Cited by 144 publications

References 11 publications

Research on the Parallelization of the DBSCAN Clustering Algorithm for Spatial Data Mining Based on the Spark Platform

Research on the Parallelization of the DBSCAN Clustering Algorithm for Spatial Data Mining Based on the Spark Platform

Distributed Gaussian Mixture Model Summarization Using the MapReduce Framework

Big Data Clustering

Contact Info

Product

Resources

About