DACA: Distributed adaptive grid decision graph based clustering algorithm

Abstract

Clustering algorithms play a very important role in machine learning. With the development of big-data artificial intelligence, distributed parallel algorithms have become an important research field. To reduce the computational complexity and running time of large-scale datasets in the clustering process, this study proposes a distributed clustering algorithm DACA (distributed adaptive grid decision graph based clustering algorithm). In a distributed environment, DACA uses relative entropy to adaptively mesh the data to form an obvious sparse grid and dense grid. Then, the decision graph is used to determine the cluster center mesh object. Finally, the KD-tree is used to accelerate the determination of the cluster center of sparse points to complete clustering. The algorithm is implemented using the popular Apache Spark computing framework, compared with other distributed clustering algorithms, DACA can adaptively divide the grid according to the data distribution to obtain better clustering effect. At the same time, KD tree algorithm is used to speed up the decision-making of clustering center. Numerous experiments show that the DACA algorithm has excellent performance and accuracy on six standard datasets and real GPS trajectory datasets. K E Y W O R D Sadaptive grid division, clustering algorithms, decision graphs, distributed, KD-tree INTRODUCTIONRecently, with the rapid development of the internet, the amount of data generated each year has been exponentially increasing. The global data volume could reach 1.8 ZB in 2 days (one of Zettabyte is 2 70 byte) 1 in 2014. By 2021, the growth of data generation will become even higher. Thus, the conventional single application-driven internet services have to be converted into data-and application-driven internet services. Therefore, big-data artificial intelligence and machine learning have been rapidly developing, the most obvious of which is the research on big data clustering algorithms, which is often discussed in the fields of internet business and academic research. 2 Clustering algorithms, which are a type of important algorithms used for data mining, have been widely employed for data analysis, image processing, market research, user segmentation, web document classification, and other fields. At present, clustering algorithms can be roughly divided into three types-traditional clustering, intelligent