Towards Improving MapReduce Task Scheduling Using Online Simulation Based Predictions

Abstract

Abstract-MapReduce is the model of choice for processing emerging big-data applications, and is facing an ever increasing demand for higher efficiency. In this context, we propose a novel task scheduling scheme that uses current task and system state information to drive online simulations concurrently within Hadoop, and predict with high accuracy future events, e.g., when a job would complete, or when task-specific datalocal nodes would be available. These predictions can then be used to make more efficient resource scheduling decisions. Our framework consists of two components: (i) Task Predictor that predicts task-level execution times based on historical data of the same type of tasks; and (ii) Job Simulator that instantiates the real task scheduler in a simulated environment, and predicts expected scheduling decisions for all the tasks comprising a MapReduce job. Evaluation shows that our framework can achieve high prediction accuracy -95% of the predicted task execution times are within 10% of the actual times -with negligible overhead (1.29%). Finally, we also present two realistic usecases, job data prefetching and a multistrategy dynamic scheduler, which can benefit from integration of our prediction framework in Hadoop.

An Improved Job Scheduling Algorithm by Utilizing Released Resources for MapReduce

“…A technique, which decides the execution order of the tasks in these slots, is the Job Scheduling Algorithms. Different Job Scheduling Algorithms [3][4][5][6][7][8][9][10][11][12], [15], [19][20][21][22][23], [25] are used for this parallel processing framework which takes care of the Data Locality [7][8], [24], Resource Utilization [5], MapReduce Interdependence [5] and meeting Job Deadline [9][10]. Data Locality needs to be considered when jobs are relatively smaller and data travelling and network cost cannot be ignored during the calculation of a Map Task Completion Time (MTCT).…”