A Throughput Driven Task Scheduler for Improving MapReduce Performance in Job-Intensive Environments

Wang, Xite; Shen, Derong; Yu, Ge; Nie, Tiezheng; Kou, Yue

doi:10.1109/bigdata.congress.2013.36

Cited by 9 publications

(4 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sarood [34] proposed a model to distribute available nodes and power amongst the queued jobs such that the throughput of HPC data centers is maximized under a given power budget. Numerous job scheduling algorithms [3, 39,38,36] have been proposed with various optimization objectives including system efficiency and throughput. These approaches assume that once scheduled, all jobs run at 100% efficiency.…”

Section: Discussion and Related Workmentioning

confidence: 99%

How Much SSD Is Useful for Resilience in Supercomputers

Fang

Chien

2015

Proceedings of the 5th Workshop on Fault Tolerance for HPC at eXtreme Scale

View full text Add to dashboard Cite

We consider the use of non-volatile memories in the form of burst buffers for resilience in supercomputers. Their cost and limited lifetime demand effective use and appropriate provisioning. We develop an analytic model for the behavior of workloads on systems with burst buffers, and use it to explore questions of cost-effective provisioning, and missiondirected allocation of burst-buffer (SSD) lifetime.First, our results show that system efficiency can be increased by as much as 14% by considering a global perspective (workload mix, job size) for SSD lifetime allocation. Second, with size-based and system-efficiency based lifetime allocation, large jobs suffer as much as 40% job efficiency loss; job-efficiency based allocation must increase their allocations by 50% to eliminate this disparity. Finally, further results suggest that underprovisioning SSD lifetime (only 10-20% of the "optimum" as defined by per-job requirements without resource constraint) is sufficient to produce 90% system efficiency at failure rates three times that of current systems.

show abstract

Section: Discussion and Related Workmentioning

confidence: 99%

How Much SSD Is Useful for Resilience in Supercomputers

Fang

Chien

2015

Proceedings of the 5th Workshop on Fault Tolerance for HPC at eXtreme Scale

View full text Add to dashboard Cite

show abstract

“…Then it checks the feasibility of allocating map task i m to slot j m and reduce task i r to slot j r by checking the total processing time of the tasks against the deadline D (lines [21][22]. If the assignment of map task i m and reduce task i r is feasible (line 24), the algorithm continues to select tasks from T m and T r , and updates the variables accordingly (lines [25][26][27][28][29][30]. To keep the assignments of the tasks in alignment with the ratio of processing time f , the algorithm balances the assignment.…”

Section: Energy-aware Mapreduce Scheduling Algorithmmentioning

confidence: 99%

“…However, they did not focus on energy and power consumption as the performance criteria. Wang et al [26] proposed a task scheduling technique for MapReduce that improves the system throughput in job-intensive environments without considering the energy consumption. However, none of the above frameworks and systems exploit the job profiling information when making the decisions for task placement on the nodes to increase the energy efficiency of executing MapReduce jobs.…”

Section: Introductionmentioning

confidence: 99%

Energy-Aware Scheduling of MapReduce Jobs

Mashayekhy

Nejad

Grosu

et al. 2014

2014 IEEE International Congress on Big Data

View full text Add to dashboard Cite

The majority of large-scale data intensive applications executed by data centers are based on MapReduce or its open-source implementation, Hadoop. Such applications are executed on large clusters requiring large amounts of energy, making the energy costs a large fraction of the data center's overall costs. Therefore minimizing the energy consumption when executing MapReduce jobs is a critical concern for data centers. In this paper, we propose a framework for improving the energy efficiency of MapReduce applications, while satisfying the service level agreement (SLA). We first model the problem of energy-aware scheduling of MapReduce jobs as an Integer Program. We then propose a greedy algorithm, called Energy-aware MapReduce Scheduling Algorithm (EMRSA), that finds the assignments of map and reduce tasks to the machine slots in order to minimize the energy consumed when executing the application. We perform experiments on a large Hadoop cluster to determine the energy consumption of several MapReduce benchmark applications, and then use this data in an extensive simulation study to characterize the performance of the proposed algorithm. The results show that EMRSA is able to find job schedules consuming 40% less energy on average than the schedules obtained by a common practice scheduler that minimizes the makespan.

show abstract

“…The X. Wang, D. Shen, et al evolved Throughput Driven Task Scheduler algorithm [51]. In view of the fact that, task scheduling in MapReduce is very vital for the job execution and has a marked influence on the system performance.…”

Section: Throughput Driven Task Scheduling Algorithmmentioning

confidence: 99%

The Ambient Scrutinize of Scheduling Algorithms in Big Data Territory.

Perwej¹

2018

IJAR

View full text Add to dashboard Cite

A Throughput Driven Task Scheduler for Improving MapReduce Performance in Job-Intensive Environments

Cited by 9 publications

References 11 publications

How Much SSD Is Useful for Resilience in Supercomputers

How Much SSD Is Useful for Resilience in Supercomputers

Energy-Aware Scheduling of MapReduce Jobs

The Ambient Scrutinize of Scheduling Algorithms in Big Data Territory.

Contact Info

Product

Resources

About