Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark

Azhir, Elham; Hosseinzadeh, Mehdi; Khan, Faheem; Mosavi, Amir

doi:10.31219/osf.io/mgpr7

2022

DOI: 10.31219/osf.io/mgpr7

|View full text |Cite

Preprint

Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark

Elham Azhir¹,

Mehdi Hosseinzadeh²,

Faheem Khan³

et al.

Abstract: Access plan recommendation is a query optimization approach that executes new queries using prior created query execution plans (QEPs). The query optimizer divides the query space into clusters in the mentioned method. However, traditional clustering algorithms take a significant amount of execution time for clustering such large datasets. The MapReduce distributed computing model provides efficient solutions for storing and processing vast quantities of data. Apache Spark and Apache Hadoop frameworks are used… Show more

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2023

2024

Publication Types

Select...

Article5

Relationship

Self Cite0

Independent5

Authors

Journals

Cited by 5 publications

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

A Comprehensive Survey of Recommender Systems Based on Deep Learning

Zhou,

Xiong,

Chen

2023

Applied Sciences

View full text Add to dashboard Cite

With the increasing abundance of information resources and the development of deep learning techniques, recommender systems (RSs) based on deep learning have gradually become a research focus. Although RSs have evolved in recent years, a systematic review of existing RS approaches is still warranted. The main focus of this paper is on recommendation models that incorporate deep learning techniques. The objective is to guide novice researchers interested in this field through the investigation and application of the proposed recommendation models. Specifically, we first categorize existing RS approaches into four types: content-based recommendations, sequence recommendations, cross-domain recommendations, and social recommendation methods. We then introduce the definitions and address the challenges associated with these RS methodologies. Subsequently, we propose a comprehensive categorization framework and novel taxonomies for these methodologies, providing a thorough account of their research advancements. Finally, we discuss future developments regarding this topic.

show abstract

A Comprehensive Survey of Recommender Systems Based on Deep Learning

Zhou,

Xiong,

Chen

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

A Novel Multi-Task Performance Prediction Model for Spark

Shen,

Chen,

Rao

2023

Applied Sciences

View full text Add to dashboard Cite

Performance prediction of Spark plays a vital role in cluster resource management and system efficiency improvement. The performance of Spark is affected by several variables, such as the size of the input data, the computational power of the system, and the complexity of the algorithm. At the same time, less research has focused on multi-task performance prediction models for Spark. To address these challenges, we propose a multi-task Spark performance prediction model. The model integrates a multi-head attention mechanism and a convolutional neural network. It implements the prediction of execution times for single or multiple Spark applications. Firstly, the data are dimensionally reduced by a dimensionality reduction algorithm and fed into the model. Secondly, the model integrates a multi-head attention mechanism and a convolutional neural network. It captures complex relationships between data features and uses these features for Spark performance prediction. Finally, we use residual connections to prevent overfitting. To validate the performance of the model, we conducted experiments on four Spark benchmark applications. Compared to the benchmark prediction model, our model obtains better performance metrics. In addition, our model predicts multiple Spark benchmark applications simultaneously and maintains deviations within permissible limits. It provides a novel way for the assessment and optimization of Spark.

show abstract

JQPro:Join Query Processing in a Distributed System for Big RDF Data Using the Hash-Merge Join Technique

et al. 2023

View full text Add to dashboard Cite

In the last decade, the volume of semantic data has increased exponentially, with the number of Resource Description Framework (RDF) datasets exceeding trillions of triples in RDF repositories. Hence, the size of RDF datasets continues to grow. However, with the increasing number of RDF triples, complex multiple RDF queries are becoming a significant demand. Sometimes, such complex queries produce many common sub-expressions in a single query or over multiple queries running as a batch. In addition, it is also difficult to minimize the number of RDF queries and processing time for a large amount of related data in a typical distributed environment encounter. To address this complication, we introduce a join query processing model for big RDF data, called JQPro. By adopting a MapReduce framework in JQPro, we developed three new algorithms, which are hash-join, sort-merge, and enhanced MapReduce-join for join query processing of RDF data. Based on an experiment conducted, the result showed that the JQPro model outperformed the two popular algorithms, gStore and RDF-3X, with respect to the average execution time. Furthermore, the JQPro model was also tested against RDF-3X, RDFox, and PARJs using the LUBM benchmark. The result showed that the JQPro model had better performance in comparison with the other models. In conclusion, the findings showed that JQPro achieved improved performance with 87.77% in terms of execution time. Hence, in comparison with the selected models, JQPro performs better.

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

hi@scite.ai

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark

Cited by 5 publications

References 17 publications

A Comprehensive Survey of Recommender Systems Based on Deep Learning

A Comprehensive Survey of Recommender Systems Based on Deep Learning

A Novel Multi-Task Performance Prediction Model for Spark

JQPro:Join Query Processing in a Distributed System for Big RDF Data Using the Hash-Merge Join Technique

Contact Info

Product

Resources

About