Performance Evaluation of Spark SQL for Batch Processing

Anusha, K.; Rani, K. Usha

doi:10.1007/978-981-15-0135-7_13

Cited by 8 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A comparative analysis of the proposed work and existing works of [7] and [8] leads concludes that Spark is the favorable option of the three.…”

Section: Resultsmentioning

confidence: 98%

See 1 more Smart Citation

Aadhaar Data Analysis Comparison in MapReduce, Hive and Spark

R¹,

Ryali²,

Shrivastava³

et al. 2021

Atlantis Highlights in Computer Sciences

View full text Add to dashboard Cite

Aadhaar with a 12-digit unique identification number of every Indian provides demographic and biometric information and is mandatory for various purposes like benefit transfer directly, healthcare, etc. Approximately Aadhaar details need to store 1.3 Billion Indians which attributes to the concept of big data. In this paper, the proposed hybrid model analyses the Aadhaar dataset w.r.t different research interrogations such as count of applicants based on gender, state-wise approved and by age type applicants. In the existing systems, Aadhaar data analyses are done either manually or in primitive SQL platforms which may take days to complete. In this paper, the focus is on Aadhaar data analysis using different distributed computing frameworks like MapReduce, Hive, and Apache Spark on top of Hadoop that could be used for the purpose of better decision-making by all government firms and we provide the valid conclusion that Apache Spark framework is efficient in terms of performance.

show abstract

“…A comparative analysis of the proposed work and existing works of [7] and [8] leads concludes that Spark is the favorable option of the three.…”

Section: Resultsmentioning

confidence: 98%

“…In [7], authors uploaded word count and Transport functionality at 18 parameter values by replacing the default set. To investigate performance, they used the trial and error method to fix these components that make up the test number in a nine-node cluster with a capacity of 600 GB databases.…”

Section: Resultsmentioning

confidence: 99%

Aadhaar Data Analysis Comparison in MapReduce, Hive and Spark

R¹,

Ryali²,

Shrivastava³

et al. 2021

Atlantis Highlights in Computer Sciences

View full text Add to dashboard Cite

show abstract

“…Semantic web data are commonly represented in either RDF or RDF graph formats (Usha Rani and Lakshmi, 2020). RDF-based semantic information is shown as directed graphs, and the critical ability of RDF is to merge the data sources without having any schema definitions (Anusha and Usha Rani, 2020). It is possible to combine unstructured and semi-structured data across data websites; in such cases, users can easily crawl the data for application usage.…”

Section: Related Workmentioning

confidence: 99%

Improving the performance of query processing using proposed resilient distributed processing technique

Lakshmi

Usharani

2021

IJICC

Self Cite

View full text Add to dashboard Cite

PurposeResilient distributed processing technique (RDPT), in which mapper and reducer are simplified with the Spark contexts and support distributed parallel query processing.Design/methodology/approachThe proposed work is implemented with Pig Latin with Spark contexts to develop query processing in a distributed environment.FindingsQuery processing in Hadoop influences the distributed processing with the MapReduce model. MapReduce caters to the works on different nodes with the implementation of complex mappers and reducers. Its results are valid for some extent size of the data.Originality/valuePig supports the required parallel processing framework with the following constructs during the processing of queries: FOREACH; FLATTEN; COGROUP.

show abstract

“…In our previous work we proposed Spark-Cassandra integration and Spark SQL-Cassandra integration for Batch Processing to evaluate the performance of both frameworks on batch data [13,14]. The results of two integrated methods are compared and hence observed that Spark SQL Cassandra Integration has better Processing Performance on batch data.…”

Section: Spark Cassandra Integrationmentioning

confidence: 99%

Performance Analysis of Apache Spark MLlib Clustering on Batch Data Stored in Cassandra

Anusha

Usharani

2020

Learning and Analytics in Intelligent Systems

Self Cite

View full text Add to dashboard Cite

With the tremendous increase in the amount of data being generated from variety of sources there is a need of efficient data storage and processing techniques. Some of the sources generating this large amount of data are Weather Sensors, Scientific experiments, etc. This huge voluminous data is termed as BigData. Due to ever-increasing amount of data there is a demand for faster data ingestion and processing. Apache Spark, a dominant processing tool is a publicly available platform for processing outsized data and is mostly intended for iterative machine learning jobs. In this study, an integrated approach i.e., Spark MLlib Clustering on batch weather data stored in Cassandra database is proposed. This helps to analyze our data into number of Clusters which is required and useful for further examination of data. The main idea of this study is to evaluate Batch Processing performance of an integrated approach with two popular clustering algorithms.

show abstract

Performance Evaluation of Spark SQL for Batch Processing

Cited by 8 publications

References 0 publications

Aadhaar Data Analysis Comparison in MapReduce, Hive and Spark

Aadhaar Data Analysis Comparison in MapReduce, Hive and Spark

Improving the performance of query processing using proposed resilient distributed processing technique

Performance Analysis of Apache Spark MLlib Clustering on Batch Data Stored in Cassandra

Contact Info

Product

Resources

About