A Survey on Spark Ecosystem: Big Data Processing Infrastructure, Machine Learning, and Applications

Tang, Shanjiang; He, Bingsheng; Yu, Ce; Li, Yusen; Li, Kun

doi:10.1109/tkde.2020.2975652

Cited by 35 publications

(21 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another challenge is that memory becomes a bottleneck when dealing with a large amount of data. Keeping the data in memory is very expensive, and additional memory resources are required when a massive amount of data must be processed in computation [165].…”

Section: H Distributed Processing Of Ddos Attacks Using Hadoop and Smentioning

confidence: 99%

Machine Learning Approaches for Combating Distributed Denial of Service Attacks in Modern Networking Environments

Aljuhani

2021

IEEE Access

View full text Add to dashboard Cite

Section: H Distributed Processing Of Ddos Attacks Using Hadoop and Smentioning

confidence: 99%

Machine Learning Approaches for Combating Distributed Denial of Service Attacks in Modern Networking Environments

Aljuhani

2021

IEEE Access

View full text Add to dashboard Cite

“…Spark is one of the most commonly used big data computing platforms, it is a parallel computing framework for big data based on memory computing, which can be used to build faster and more efficient big data analysis applications [29]. Fig.…”

Section: Overview Of Spark Computing Frameworkmentioning

confidence: 99%

A Novel Bearing Fault Diagnosis Method Using Spark-Based Parallel ACO-K-Means Clustering Algorithm

Wan

Zhang

et al. 2021

IEEE Access

View full text Add to dashboard Cite

K-Means clustering algorithm is a typical unsupervised learning method, and it has been widely used in the field of fault diagnosis. However, the traditional serial K-Means clustering algorithm is difficult to efficiently and accurately perform clustering analysis on the massive running-state monitoring data of rolling bearing. Therefore, a novel fault diagnosis method of rolling bearing using Spark-based parallel ant colony optimization (ACO)-K-Means clustering algorithm is proposed. Firstly, a Spark-based three-layer wavelet packet decomposition approach is developed to efficiently preprocess the running-state monitoring data to obtain eigenvectors, which are stored in Hadoop Distributed File System (HDFS) and served as the input of ACO-K-Means clustering algorithm. Secondly, ACO-K-Means clustering algorithm suitable for rolling bearing fault diagnosis is proposed to improve the diagnosis accuracy. ACO algorithm is adopted to obtain the global optimal initial clustering centers of K-Means from all eigenvectors, and the K-Means clustering algorithm based on weighted Euclidean distance is used to perform clustering analysis on all eigenvectors to obtain a rolling bearing fault diagnosis model. Thirdly, the efficient parallelization of ACO-K-Means clustering algorithm is implemented on a Spark platform, which can make full use of the computing resources of a cluster to efficiently process large-scale rolling bearing datasets in parallel. Extensive experiments are conducted to verify the effectiveness of the proposed fault diagnosis method. Experimental results show that the proposed method can not only achieve good fault diagnosis accuracy but also provide high model training efficiency and fault diagnosis efficiency in a big data environment.

show abstract

“…Furthermore, each RDD is divided into several partitions that can be calculated on the various nodes of the cluster, allowing for data distribution and parallelization. In Spark, all functions are carried out on RDDs [24].…”

Section: Related Workmentioning

confidence: 99%

“…In fact, the big satellite image data is incorporated in strips into the RDDs [24]. Additionally, using HDFS the couple key/value can be efficiently generated to the image partitions in order to conceal the remote sensing data heterogeneity.…”

Section: Related Workmentioning

confidence: 99%

Big Remote Sensing Image Classification Based on Deep Learning Extraction Features and Distributed Spark Frameworks

Chebbi

Mellouli²,

Farah

et al. 2021

BDCC

View full text Add to dashboard Cite

Big data analysis assumes a significant role in Earth observation using remote sensing images, since the explosion of data images from multiple sensors is used in several fields. The traditional data analysis techniques have different limitations on storing and processing massive volumes of data. Besides, big remote sensing data analytics demand sophisticated algorithms based on specific techniques to store to process the data in real-time or in near real-time with high accuracy, efficiency, and high speed. In this paper, we present a method for storing a huge number of heterogeneous satellite images based on Hadoop distributed file system (HDFS) and Apache Spark. We also present how deep learning algorithms such as VGGNet and UNet can be beneficial to big remote sensing data processing for feature extraction and classification. The obtained results prove that our approach outperforms other methods.

show abstract

A Survey on Spark Ecosystem: Big Data Processing Infrastructure, Machine Learning, and Applications

Cited by 35 publications

References 72 publications

Machine Learning Approaches for Combating Distributed Denial of Service Attacks in Modern Networking Environments

Machine Learning Approaches for Combating Distributed Denial of Service Attacks in Modern Networking Environments

A Novel Bearing Fault Diagnosis Method Using Spark-Based Parallel ACO-K-Means Clustering Algorithm

Big Remote Sensing Image Classification Based on Deep Learning Extraction Features and Distributed Spark Frameworks

Contact Info

Product

Resources

About