Big Data and its Analyzing Tools : A Perspective

Jaiswal, Ayshwarya; Dwivedi, Vijay Kumar; Yadav, Om Prakash

doi:10.1109/icaccs48705.2020.9074222

Cited by 7 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Apache Spark is a revolutionary framework that can perform in-memory computing faster than Hadoop MapReduce for large-scale preprocessing. Compared with Hadoop MapReduce, Spark [2], [4] is characterized by fast computation speed, ease of use, high generality, and run everywhere. Spark can cache the intermediate data into memory during computation and can persist data into disk when memory overflows.…”

Section: Spark Frameworkmentioning

confidence: 99%

See 1 more Smart Citation

A Novel Spark-Based Attribute Reduction and Neighborhood Classification for Rough Evidence

Ding

Sun

et al. 2024

IEEE Trans. Cybern.

View full text Add to dashboard Cite

Neighborhood classification (NEC) algorithms has been widely used to solve classification problems. Most traditional NEC algorithms employ the majority voting mechanism as the basis for final decision-making. However, this mechanism hardly considers the spatial difference and label uncertainty of the neighborhood samples, which may increase the possibility of the misclassification. In addition, the traditional NEC algorithms need to load the whole data into memory at once, which is computationally inefficient when the size of data set is large. To address these problems, we propose a novel Spark-based attribute reduction and neighborhood classification for rough evidence in this paper. Specifically, we first construct a multi-granular sample space using parallel undersampling method. Then, we evaluate the significance of attribute by neighborhood rough evidence decision error rate and remove the redundant attribute on different samples subspaces. Based on this attribute reduction algorithm, we design a parallel attribute reduction algorithm which is able to compute equivalence classes in parallel and parallelize the process of searching for candidate attributes. Finally, we introduce the rough evidence into the classification decision of traditional NEC algorithms and parallelize the classification decision process. Furthermore, the proposed algorithms are conducted in the Spark parallel computing framework. Experimental results on both small and large-scale data sets show that the proposed algorithms outperform the benchmarking algorithms in the classification accuracy and the computational efficiency.

show abstract

Section: Spark Frameworkmentioning

confidence: 99%

“…caused serious challenges to traditional data mining and analysis technologies, and brought valuable opportunities to the development of various industries [1], [2]. Because of the huge scale and low value density of big data, it has brought several challenges to traditional data mining techniques.…”

mentioning

confidence: 99%

A Novel Spark-Based Attribute Reduction and Neighborhood Classification for Rough Evidence

Ding

Sun

et al. 2024

IEEE Trans. Cybern.

View full text Add to dashboard Cite

show abstract

“… Latency [24], [33], [36]  Apache Hadoop: Due to its support for various data types, structures, and volumes, Hadoop's MapReduce framework is comparatively slower than other frameworks. Hadoophas a higher latency than Spark and Flink because of this.…”

Section:  Language Support [2] [17] [34] [35]mentioning

confidence: 99%

A Comprehensive Study on Big Data Frameworks

Sultan¹,

Abdullah²

2023

MINAR Journal

View full text Add to dashboard Cite

With the advent of cloud computing technology, the generation of data from various sources has increased during the last few years. The current data processing technology must handle the enormous volumes of newly created data. Therefore, the studies in the literature have concentrated on big data, which has enormous volumes of almost unstructured data. Dealing with such data needs well-designed frameworks that fulfil developers’ needs and fit colourful purposes. Moreover, these frameworks can use for storing, processing, structuring, and analyzing data. The main problem facing cloud computing developers is selecting the most suitable framework for their applications. The literature includes many works on these frameworks. However, there is still a severe gap in providing comprehensive studies on this crucial area of research. Hence, this article presents a novel comprehensive comparison among the most popular frameworks for big data, such as Apache Hadoop, Apache Spark, Apache Flink, Apache Storm, and MongoDB. In addition, the main characteristics of each framework in terms of advantages and drawbacks are also deeply investigated in this article. Our research provides a comprehensive analysis of various metrics related to data processing, including data flow, computational model, overall performance, fault tolerance, scalability, interval processing, language support, latency, and processing speed. To our knowledge, no previous research has conducted a detailed study of all these characteristics simultaneously. Therefore, our study contributes significantly to the understanding of the factors that impact data processing and provides valuable insights for practitioners and researchers in the field

show abstract

“…Speed, powerful caching, deployment, real-time support, and multilingual support are some of the benefits of Spark. Using controlled partitioning, Spark speeds up to 100 times quicker than Hadoop Map/Reduce (Jaiswal et al, 2020). MongoDB is a popular NoSQL database that uses a document-oriented data model.…”

Section: Introductionmentioning

confidence: 99%

Investigating scientific collaboration networks in Iraq using cloud computing and data mining

Sultan,

Abdullah

2023

Int. J. Appl. Sci. Eng.

View full text Add to dashboard Cite

The academic performance of Iraqi authors and institutions is an important aspect that needs to be investigated in the field of research. Special scientific criteria establish a researcher's level. These include citations, published articles, and others. In this study, the aim is to analyze the academic performance patterns of Iraqi authors and institutions in Google Scholar. Collaboration patterns between Iraqi scientists and universities were explored using sophisticated data collection and analysis techniques. A crawler that worked in parallel with Google Scholar was created with SerpApi, thus collecting amounts of scientist profile data stored in MongoDB Atlas. The data were processed using Spark within the Amazon Web Services (AWS) Cloud. Complex network analysis and Cytoscape software were used to comprehensively investigate the co-authorship networks of Iraqi scientists. In addition, a set of complex network criteria was applied to reveal patterns of cooperation. After visualizing the network diagram and analyzing the relationships between the nodes, the virtual nodes and communities in the network were identified, revealing critical insights into the cooperation patterns of Iraqi scientists and universities. The results have important implications for research and development policies in Iraq and demonstrate the power of complex network analysis to reveal valuable insights into academic research collaboration.

show abstract

Big Data and its Analyzing Tools : A Perspective

Cited by 7 publications

References 8 publications

A Novel Spark-Based Attribute Reduction and Neighborhood Classification for Rough Evidence

A Novel Spark-Based Attribute Reduction and Neighborhood Classification for Rough Evidence

A Comprehensive Study on Big Data Frameworks

Investigating scientific collaboration networks in Iraq using cloud computing and data mining

Contact Info

Product

Resources

About