Analysis of hadoop MapReduce scheduling in heterogeneous environment

Kalia, Khushboo; Gupta, Neeraj

doi:10.1016/j.asej.2020.06.009

Cited by 54 publications

(19 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The entire mapping and reducing processes are implemented in parallel and the size of huge data is minimized in reduction processes. Hadoop MapReduce 37 constitutes of job and task trackers. The function of the job tracker is to split the MapReduce tasks into small tasks in which the task trackers process the tasks.…”

Section: Proposed Methodologymentioning

confidence: 99%

A novel deep neural network based marine predator model for effective classification of big data from social Internet of Things

Shaji

Singh

Nisha

2022

Concurrency and Computation

View full text Add to dashboard Cite

Social Internet of Things (SIoT) is considered as one of the most recently evolved topics that connects people and object, object and object as well as people and people. The SIoT and big data provide an exact representation of IoT and social system for human progression characterization. Numerous machine learning techniques are employed to classify the data gathered from SIoT in a more powerful way. In this article, a deep neural network based marine predator (DRNN-MP) algorithm is proposed in classifying big data. Here, an adaptive Savitzky-Golay filter is employed for selecting the subset and to eliminate undesirable data, as well as different noises. The big data databases are reduced using a Hadoop map reducing framework thereby enhancing the performances of the proposed approach. In addition to this, a modified relief technique is employed to select optimal features thereby performing better classification. The testing and training process based on the proposed approach for optimal classification of features from the big data considers four different databases namely coronary illness, GPS trajectories, localization data, and water treatment plant obtained from UCI machine learning repository. In addition to this, the proposed approach is evaluated for diverse performance measures namely accuracy, precision, specificity, sensitivity, throughput, and energy consumption. Finally, the proposed approach is compared for various metrics to illustrate the effectiveness of the system and the results demonstrated that the accuracy of our work is 98.25%. K E Y W O R D Sbig data, database, deep recurrent neural network, marine predator, modified relief, social internet of things INTRODUCTIONInternet of things (IoT) illustrates a separate universe of heterogeneous objects namely actuators, sensors as well as smartphones in which each object and everything possesses a sovereign identity. 1 Nowadays, developing an IoT enabled technology and providing solutions are considered as tedious tasks. On the other hand, IoT deals with sharing and pervasive collection of data regarding a common intention. In IoT, various attribute factors namely integer value and variable define the data thereby permitting specific functions to be performed using a predetermined interface. 2Numerous research scholars are fascinated peculiarly in determining the risk problems emerging while integrating and discovering the data withinIoT. In recent years, the social Internet of Things (SIoT) has been established which is another utilization of IoT. 3 They constitute various distinctive characteristics including diverse communication protocols, operating systems, related standards, and various operational platforms; however, all such differences are neglected. 4

show abstract

Section: Proposed Methodologymentioning

confidence: 99%

A novel deep neural network based marine predator model for effective classification of big data from social Internet of Things

Shaji

Singh

Nisha

2022

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…There are several surveys [3][4][5][6][7] of existing job schedulers that discuss their features, advantages, and limitations. They have classified job schedulers based on different aspects: strategy (static/dynamic), environment (homogenous/heterogeneous), time (deadline/ delay), etc.…”

Section: Related Workmentioning

confidence: 99%

CLQLMRS: improving cache locality in MapReduce job scheduling using Q-learning

et al. 2022

View full text Add to dashboard Cite

Scheduling of MapReduce jobs is an integral part of Hadoop and effective job scheduling has a direct impact on Hadoop performance. Data locality is one of the most important factors to be considered in order to improve efficiency, as it affects data transmission through the system. A number of researchers have suggested approaches for improving data locality, but few have considered cache locality. In this paper, we present a state-of-the-art job scheduler, CLQLMRS (Cache Locality with Q-Learning in MapReduce Scheduler) for improving both data locality and cache locality using reinforcement learning. The proposed algorithm is evaluated by various experiments in a heterogeneous environment. Experimental results show significantly decreased execution time compared with FIFO, Delay, and the Adaptive Cache Local scheduler.

show abstract

“…Then, they used roulette wheel scheme (RWS) based data block placement on heterogeneous VMs to improve makespan and resource utilization. Various scheduling schemes for Hadoop MapReduce in a heterogeneous environment are discussed in [12]. The authors focused on how the heterogeneous environment affects the performance in MapReduce job execution sequence.…”

Section: Literature Surveymentioning

confidence: 99%

Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization

Jeyaraj

Paul

2022

IEEE Access

View full text Add to dashboard Cite

Consuming Hadoop MapReduce via virtual infrastructure as a service is becoming common practice as cloud service providers (CSP) offers relevant applications and scalable resources. One of the predominant requirements of cloud users is to improve resource utilization in the virtual cluster during the service period. However, it may not be possible when MapReduce workloads and virtual machines (VM) are highly heterogeneous. Therefore, in this paper, we addressed these heterogeneities and proposed an efficient MapReduce scheduler to improve resource utilization by placing the right combination of the map and reduce tasks in each VM in the virtual cluster. To achieve this, we transformed the MapReduce task scheduling problem into a 2-Dimensional (2D) bin packing model and obtained an optimal schedule using the ant colony optimization (ACO) algorithm. As an added advantage, our proposed ACO based bin packing (ACO-BP) scheduler minimized the makespan for a batch of jobs. To showcase the performance improvement, we compared our proposed scheduler with three existing schedulers that work well in a heterogeneous environment. As expected, results show that ACO-BP significantly outperformed the existing schedulers while dealing with workload and VM level heterogeneities.

show abstract

Analysis of hadoop MapReduce scheduling in heterogeneous environment

Cited by 54 publications

References 32 publications

A novel deep neural network based marine predator model for effective classification of big data from social Internet of Things

A novel deep neural network based marine predator model for effective classification of big data from social Internet of Things

CLQLMRS: improving cache locality in MapReduce job scheduling using Q-learning

Optimizing MapReduce Task Scheduling on Virtualized Heterogeneous Environments Using Ant Colony Optimization

Contact Info

Product

Resources

About