A Hadoop Job Scheduling Model Based on Uncategorized Slot

Abstract

Job scheduling is becoming an important part of Hadoop framework at present, a job scheduling model based on uncategorized slot was researched on this paper. It could eliminate the limitation of Job Task type and didn't distinguish between Map slot and Reduce slot any more, however there was only one type of slot left which could be assigned to execute the Map tasks and to run the Reduce tasks. By adopting Reduce dynamic partitioning, it can realize switching smoothly the slot between two types of tasks, meanwhile, compared with the FIFO algorithm which need distinguish the type of slot, the experimental result shows that the model not only improves the resources utilization and betters load balancing, but also enhances the parallelism of tasks and shortens the execution time of the Job Tasks.