A novel approach for big data classification based on hybrid parallel dimensionality reduction using spark cluster

Abstract

The big data concept has elicited studies on how to accurately and efficiently extract valuable information from such huge dataset. The major problem during big data mining is data dimensionality due to a large number of dimensions in such datasets. This major consequence of high data dimensionality is that it affects the accuracy of machine learning (ML) classifiers; it also results in time wastage due to the presence of several redundant features in the dataset. This problem can be possibly solved using a fast feature reduction method. Hence, this study presents a fast HP-PL which is a new hybrid parallel feature reduction framework that utilizes spark to facilitate feature reduction on shared/distributed-memory clusters. The evaluation of the proposed HP-PL on KDD99 dataset showed the algorithm to be significantly faster than the conventional feature reduction techniques. The proposed technique required 1 minute to select 4 dataset features from over 79 features and 3,000,000 samples on a 3-node cluster (total of 21 cores). For the comparative algorithm, more than 2 hours was required to achieve the same feat. In the proposed system, Hadoop’s distributed file system (HDFS) was used to achieve distributed storage while Apache Spark was used as the computing engine. The model development was based on a parallel model with full consideration of the high performance and throughput of distributed computing. Conclusively, the proposed HP-PL method can achieve good accuracy with less memory and time compared to the conventional methods of feature reduction. This tool can be publicly accessed at https://github.com/ahmed/Fast-HP-PL.

Large scale data analysis using MLlib

“…A suite of multiple-language APIs is also available on the Apache Spark MLlib [117] platform for the evaluation and deployment of a wide range of machine learning techniques. In recent years, several changes have been made to multiple areas of data science solutions [118], [119], and a number of academics have committed attention to the creation of the components of Apache Spark MLlib for big data analytics.…”

Analysis of LTE physical channels overhead

“…MAC layer used to map the transport channels to physical channels and it performs scheduling functions in eNB every TTI by assigning RBs to mobile subscribers. The scheduler also determines different transmission parameters like transmitted power and modulation/coding scheme (MCS) which pointed to as radio link adaptation [7,[17][18][19][20]. LTE supports various MCS which can be changed per 1 ms subframe based on wireless channel conditions and interference.…”

Fuzzy Generalized Hebbian Algorithm for Large-Scale Intrusion Detection System

“…Hence, DT performed better in almost all the attacks. Xiang et al [9] suggested a novel linear correlation feature reduction framework. This framework is beneficial in cases of marginally unrelated features which are jointly related to the response.…”

“…Hence, there is a need to have an established database of attack signatures [7,8]. The MB approach ensures a good detection of well-known network attacks [4,9]. One major problem of the MB approach is that new or unfamiliar attacks may not be detected.…”