Job Scheduling Optimization for Multi-user MapReduce Clusters

Tao, Yongcai; Zhang, Qing; Shi, Lei; Chen, Pin-Hua

doi:10.1109/paap.2011.33

Cited by 15 publications

(5 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Sarood [34] proposed a model to distribute available nodes and power amongst the queued jobs such that the throughput of HPC data centers is maximized under a given power budget. Numerous job scheduling algorithms [3, 39,38,36] have been proposed with various optimization objectives including system efficiency and throughput. These approaches assume that once scheduled, all jobs run at 100% efficiency.…”

Section: Discussion and Related Workmentioning

confidence: 99%

How Much SSD Is Useful for Resilience in Supercomputers

Fang

Chien

2015

Proceedings of the 5th Workshop on Fault Tolerance for HPC at eXtreme Scale

View full text Add to dashboard Cite

We consider the use of non-volatile memories in the form of burst buffers for resilience in supercomputers. Their cost and limited lifetime demand effective use and appropriate provisioning. We develop an analytic model for the behavior of workloads on systems with burst buffers, and use it to explore questions of cost-effective provisioning, and missiondirected allocation of burst-buffer (SSD) lifetime.First, our results show that system efficiency can be increased by as much as 14% by considering a global perspective (workload mix, job size) for SSD lifetime allocation. Second, with size-based and system-efficiency based lifetime allocation, large jobs suffer as much as 40% job efficiency loss; job-efficiency based allocation must increase their allocations by 50% to eliminate this disparity. Finally, further results suggest that underprovisioning SSD lifetime (only 10-20% of the "optimum" as defined by per-job requirements without resource constraint) is sufficient to produce 90% system efficiency at failure rates three times that of current systems.

show abstract

Section: Discussion and Related Workmentioning

confidence: 99%

How Much SSD Is Useful for Resilience in Supercomputers

Fang

Chien

2015

Proceedings of the 5th Workshop on Fault Tolerance for HPC at eXtreme Scale

View full text Add to dashboard Cite

show abstract

“…The first MapReduce schedulers operated on First-In-First-Out (FIFO) order. Later, Fair Scheduling [24] was introduced for using max-min fairness to share resources between pools of jobs or users.…”

Section: Mapreduce and Portfolio Schedulingmentioning

confidence: 99%

POSUM: A Portfolio Scheduler for MapReduce Workloads

Voinea

Uta

Iosup

2018

2018 IEEE International Conference on Big Data (Big Data)

View full text Add to dashboard Cite

General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

show abstract

“…It manages all issues related to partitioning the input data, scheduling the program's execution and data transfers. Several research papers are focused on the MapReduce model to apply it to some business domains [10], [11] to resolve some algorithms issues [12], [13] or to search for some optimization leads [14], [15] (Fig. 4).…”

Section: B Mapreducementioning

confidence: 99%

MapReduce Programs Simplification using a Query Criteria API

Hassan¹,

Namir²,

Rachiq³

et al. 2018

ijacsa

View full text Add to dashboard Cite

A Hadoop HDFS is an organized and distributed collection of files. It is created to store a huge part of data and then retrieve it and analyze it efficiently in a less amount of time. To retrieve and analyze data from the Hadoop HDFS, MapReduce Jobs must be created directly using some programming languages like Java or indirectly using some high level languages like HiveQL and PigLatin. Everyone knows that creating MapReduce programs using programming languages is a difficult task that requires a remarkable effort for their creation and also for their maintenance. Writing MapReduce code by hand needs a lot of time, introduce bugs, harm readability, and impede optimizations. Profiles working in the field of big data always try to avoid hard and long programs in their work. They are always looking for much simpler alternatives like graphical interfaces or reduced scripts like PIG Latin or even SQL queries. This article proposes to use a MapReduce Query API inspired from Hibernate Criteria to simplify the code of MapReduce programs. This API proposes a set of predefined methods for making restrictions, projections, logical conditions and so on. An implementation of the Word Count example using the Query Criteria API is illustrated in this paper.

show abstract

Job Scheduling Optimization for Multi-user MapReduce Clusters

Cited by 15 publications

References 4 publications

How Much SSD Is Useful for Resilience in Supercomputers

How Much SSD Is Useful for Resilience in Supercomputers

POSUM: A Portfolio Scheduler for MapReduce Workloads

MapReduce Programs Simplification using a Query Criteria API

Contact Info

Product

Resources

About