Improving the performance of query processing using proposed resilient distributed processing technique

Abstract

PurposeResilient distributed processing technique (RDPT), in which mapper and reducer are simplified with the Spark contexts and support distributed parallel query processing.Design/methodology/approachThe proposed work is implemented with Pig Latin with Spark contexts to develop query processing in a distributed environment.FindingsQuery processing in Hadoop influences the distributed processing with the MapReduce model. MapReduce caters to the works on different nodes with the implementation of complex mappers and reducers. Its results are valid for some extent size of the data.Originality/valuePig supports the required parallel processing framework with the following constructs during the processing of queries: FOREACH; FLATTEN; COGROUP.

A comprehensive bibliometric analysis of Apache Hadoop from 2008 to 2020

“…With the increasing demand for data storage and computing performance, Hadoop is not sufficient to satisfy the requirements, which is designed to be deployed on low-cost hardware and is highly fault-tolerant, while some performance has been sacrificed. But the appearance of Apache Spark and Apache Flink, making up for this shortcoming to a large extent by maintaining task state in memory, greatly accelerates the speed of data processing, and makes the ecosystem of Hadoop more mature (Lakshmi and Usha Rani, 2021). Through the investigation of the published literature, the research about Apache Spark and Apache Flink is becoming popular, while research of Hadoop goes through a downward tendency from 2016.…”