Optimization of Slot and Map Reduce Workload

Abstract

MapReduce might be a modish computing paradigm in Hadoop. MapReduce is utilized for large-scale process in large info. However, the slot-based MapReduce system (e.g., Hadoop MRv1) can suffer from poor performance as a result of its unoptimized resource allocation. to resolve this downside this paper identifies and optimizes the resource allocation from three key aspects.First, alowing to the pre-configuration of distinct reduce slots and map slots that aren't fungible, slots will be severely under-utilized. as a result of map slots can be totally used whereas reduce slots area unit empty, and vice-versa. This paper conjointly proposes another technique known as Dynamic Hadoop Slot Allocation by keeping the slot-based model. It relaxes the slot allocation constraint to permit slots to be reallocated to either map or reduce tasks reckoning on their desires. Second, the speculative execution will tackle the strayer drawback, that has shown to boost the performance for one job however at the expense of the cluster potenc In view of this, we tend to propose Speculative Execution Performance leveling to balance the performance between one job and a bunch of jobs. Third, delay programming has shown to enhance the info neighbourhood however at the price of fairness. to boot, the paper propose a way referred to as Slot PreScheduling which will improve the info neighbourhood however with no impact on fairness. Finally, by combining these techniques along, we tend to type a gradual slot allocation system referred to as DynamicMR which will improve the performance of MapReduce workloads considerably.