Density-based Algorithms for Big Data Clustering Using MapReduce Framework

Khader, Mariam; Al‐Naymat, Ghazi

doi:10.1145/3403951

Cited by 13 publications

(8 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The HDFS is a scalable file system, has a high throughput, and can handle errors. Figure 1 shows the main tasks of MapReduce, which can be summed up in the following way [1], [10]: -Mappers process the input data, which is read from the HDFS file system, and produce output pairs of "key/value." -During the shuffling step, the outputs of the mappers are redirected to reducer nodes by the values of their respective "keys."…”

Section: Mapreduce Frameworkmentioning

confidence: 99%

An innovativefractal architecture model for implementing MapReduce in an open multiprocessing parallel environment

Khudhair

AL-Rammahi

Rabee

2023

IJEECS

View full text Add to dashboard Cite

One of the infrastructure applications that cloud computing offers as a service is parallel data processing. MapReduce is a type of parallel processing used more and more by data-intensive applications in cloud computing environments. MapReduce is based on a strategy called "divide and conquer," which uses regular computers, also called "nodes," to do processing in parallel. This paper looks at how open multiprocessing (OpenMP), the best shared-memory parallel programming model for high-performance computing, can be used with the proposed fractal network model in the MapReduce application. A well-known model, the cube, is used to compare the fractal network model and its work. Where experiments demonstrated that the fractal model is preferable to the cube model. The fractal model achieved an average speedup of 2.7 and an efficiency rate of 67.7%. In contrast, the cube model could only reach an average speedup of 2.5 and an efficiency rate of 60.4%.

show abstract

Section: Mapreduce Frameworkmentioning

confidence: 99%

An innovativefractal architecture model for implementing MapReduce in an open multiprocessing parallel environment

Khudhair

AL-Rammahi

Rabee

2023

IJEECS

View full text Add to dashboard Cite

show abstract

“…In ZTA, we believe that AI technologies are the primary solution for automating cyber threat intelligence collection. The clustering algorithms of unsupervised learning can group different patterns of threat intelligence according to their similarity [6,70]. The log-based anomaly detection method uses AI technologies to realize automatic log monitoring and anomaly identification.…”

Section: Threat Intelligence Collectionmentioning

confidence: 99%

Automation and Orchestration of Zero Trust Architecture:Potential Solutions and Challenges

Cao¹,

Pokhrel²,

Zhu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Zero Trust Architecture (ZTA) is a paradigm shift in how we protect data, stay connected, and access resources. ZTA is non-perimeter based defense, which has been emerging as a promising revolution in the cybersecurity field. It can be used to continuously maintain security by safeguarding against attacks both from inside and outside of the network system. However, automation and orchestration ZTA, an essential direction towards its seamless deployments over real-word networks, have been poorly understood in literature. In this paper, we first identify the bottlenecks, discuss the background of ZTA and compare it with traditional perimeter-based security architectures. More importantly, we present a comprehensive direction towards the automation and orchestration of ZTA by employing AI techniques. In other words, the key potential and roles of celebrates AI techniques for the automation and orchestration of ZTA are demonstrated for further exploration. Overall, in this review paper, we develop a foundational view on the challenges and potential enablers for the automation and orchestration of ZTA.

show abstract

“…These data mining techniques are also known as Knowledge Discovery in Database. The common ways used in mining data includes clustering [3], regression, classification, and association rule mining algorithms, among which, density-based clustering algorithm [4] as a branch of clustering is widely used in the geography, medicine, finance, and image analysis fields [5] because of its capability to discover clusters of arbitrary shapes in dense areas and handle noise or outliers effectively. Unfortunately, the huge volume of data generated in the traditional density-based clustering algorithms, can be too massive for a single machine to handle in a reasonable amount of time.…”

Section: Introductionmentioning

confidence: 99%

MR-DBIFOA: a parallel Density-based Clustering Algorithm by Using Improve Fruit Fly Optimization

Liu¹,

Liu²,

Wang³

et al. 2022

Journal of Compurters

View full text Add to dashboard Cite

Clustering is an important technique for data analysis and knowledge discovery. In the context of big data, the density-based clustering algorithm faces three challenging problems: unreasonable division of data gridding, poor parameter optimization ability and low efficiency of parallelization. In this study, a density-based clustering algorithm by using improve fruit fly optimization based on MapReduce (MR-DBIFOA) is proposed to tackle these three problems. Firstly, based on KD-Tree, a division strategy (KDG) is proposed to divide the cell of grid adaptively. Secondly, an improve fruit fly optimization algorithm (IFOA) which use the step strategy based on knowledge learn (KLSS) and the clustering criterion function (CFF) is designed. In addition, based on IFOA algorithm, the optimal parameters of local clustering are dynamically selected, which can improve the clustering effect of local clustering. Meanwhile, in order to improve the parallel efficiency, the density-based clustering algorithm using IFOA (MR-QRMEC) are proposed to parallel compute the local clusters of clustering algorithm. Finally, based on QR-Tree and MapReduce, a cluster merging algorithm (MR-QRMEC) is proposed to get the result of clustering algorithm more quickly, which improve the core clusters merging efficiency of density-based clustering algorithm. The experimental results show that the MR-DBIFOA algorithm has better clustering results and performs better parallelization in big data.

show abstract

Density-based Algorithms for Big Data Clustering Using MapReduce Framework

Cited by 13 publications

References 78 publications

An innovativefractal architecture model for implementing MapReduce in an open multiprocessing parallel environment

An innovativefractal architecture model for implementing MapReduce in an open multiprocessing parallel environment

Automation and Orchestration of Zero Trust Architecture:Potential Solutions and Challenges

MR-DBIFOA: a parallel Density-based Clustering Algorithm by Using Improve Fruit Fly Optimization

Contact Info

Product

Resources

About