Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark

Azhir, Elham; Hosseinzadeh, Mehdi; Khan, Faheem; Mosavi, Amir

doi:10.3390/math10193517

Cited by 5 publications

(3 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Venturing into the realm of natural language processing, the transformer model [82] gained widespread acclaim. It established unparalleled standards in terms of discerning the dependencies between sequences, which paved the way for swift parallel computations and accelerated sequence information extraction [83].…”

Section: Long-and Short-term Sequence Recommendationmentioning

confidence: 99%

A Comprehensive Survey of Recommender Systems Based on Deep Learning

Zhou,

Xiong,

Chen

2023

Applied Sciences

View full text Add to dashboard Cite

With the increasing abundance of information resources and the development of deep learning techniques, recommender systems (RSs) based on deep learning have gradually become a research focus. Although RSs have evolved in recent years, a systematic review of existing RS approaches is still warranted. The main focus of this paper is on recommendation models that incorporate deep learning techniques. The objective is to guide novice researchers interested in this field through the investigation and application of the proposed recommendation models. Specifically, we first categorize existing RS approaches into four types: content-based recommendations, sequence recommendations, cross-domain recommendations, and social recommendation methods. We then introduce the definitions and address the challenges associated with these RS methodologies. Subsequently, we propose a comprehensive categorization framework and novel taxonomies for these methodologies, providing a thorough account of their research advancements. Finally, we discuss future developments regarding this topic.

show abstract

Section: Long-and Short-term Sequence Recommendationmentioning

confidence: 99%

A Comprehensive Survey of Recommender Systems Based on Deep Learning

Zhou,

Xiong,

Chen

2023

Applied Sciences

View full text Add to dashboard Cite

show abstract

“…Data locality and Hadoop [10] parameter tuning in dependable and uniform cluster environments make up the majority of the related attempt for enhancing Map reduce performance. [11] Cautioned have been used to categorize this system.…”

Section: Literature Surveymentioning

confidence: 99%

An Enhanced Query Optimization Implemented in Hadoop using Bio-Inspired Algorithm with HDFS Technique

Abhijit Banubakode

2023

IJRITCC

View full text Add to dashboard Cite

A more effective method for massive data query optimization using HDFS and the Bio-inspired algorithm. Big Data configuration and query optimization are the two phases of the process. To remove redundant data, the input data is first per-processed using HDFS. Then, utilizing entropy calculation, features like closed frequent pattern, support, and confidence are extracted and managed. The Bio-inspired Horse Herd approach is used to group pertinent information based on this outcome. In the second step, the Big Data queries are used to obtain the same features. The optimized query is then located using the Bio-inspired technique, and the similarity assessment procedure is run. The proposed algorithm, according to this research, outperforms other ones that is unique in use. It is challenging to determine the veracity of this claim without more information regarding the experimental setup and the precise measures employed to assess the algorithm's effectiveness. Furthermore, it is unknown how the proposed algorithm stacks up against other cutting-edge query optimization methods. Finally, the assess has efficiency of using this method, more optimistic query achieved and comparison analysis are proved.

show abstract

“…For example, Spark can be used to run big data queries, so query-wise performance prediction is also important. Azhir et al [14] used Spark and Hadoop to cluster query datasets of various sizes and evaluate query performance. Yadav et al [15] analyzed the impact of data size on the query execution time for Spark, which is a popular big data query framework.…”

Section: Related Workmentioning

confidence: 99%

A Novel Multi-Task Performance Prediction Model for Spark

Shen,

Chen,

Rao

2023

Applied Sciences

View full text Add to dashboard Cite

Performance prediction of Spark plays a vital role in cluster resource management and system efficiency improvement. The performance of Spark is affected by several variables, such as the size of the input data, the computational power of the system, and the complexity of the algorithm. At the same time, less research has focused on multi-task performance prediction models for Spark. To address these challenges, we propose a multi-task Spark performance prediction model. The model integrates a multi-head attention mechanism and a convolutional neural network. It implements the prediction of execution times for single or multiple Spark applications. Firstly, the data are dimensionally reduced by a dimensionality reduction algorithm and fed into the model. Secondly, the model integrates a multi-head attention mechanism and a convolutional neural network. It captures complex relationships between data features and uses these features for Spark performance prediction. Finally, we use residual connections to prevent overfitting. To validate the performance of the model, we conducted experiments on four Spark benchmark applications. Compared to the benchmark prediction model, our model obtains better performance metrics. In addition, our model predicts multiple Spark benchmark applications simultaneously and maintains deviations within permissible limits. It provides a novel way for the assessment and optimization of Spark.

show abstract

Performance Evaluation of Query Plan Recommendation with Apache Hadoop and Apache Spark

Cited by 5 publications

References 21 publications

A Comprehensive Survey of Recommender Systems Based on Deep Learning

A Comprehensive Survey of Recommender Systems Based on Deep Learning

An Enhanced Query Optimization Implemented in Hadoop using Bio-Inspired Algorithm with HDFS Technique

A Novel Multi-Task Performance Prediction Model for Spark

Contact Info

Product

Resources

About