Maestro: Replica-Aware Map Scheduling for MapReduce

Ibrahim, Shadi; Jin, Hai; Lu, Lu; He, Bingsheng; Antoniu, Gabriel; Wu, Song

doi:10.1109/ccgrid.2012.122

Cited by 87 publications

(51 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Providing locality for tasks is crucial for performance of Hadoop in large clusters because network bisection bandwidth becomes a bottleneck [10] [16]. Besides, since most of the Hadoop usage is for small jobs (jobs with a small number of map tasks) [26], it is difficult for a small job to obtain slots on nodes with local data.…”

Section: Discussion On Hadoop Performance Under Failurementioning

confidence: 99%

An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop’s Schedulers Under Failures

Ibrahim

Phuong

Antoniu

2015

Adaptive Resource Management and Scheduling for Cloud Computing

Self Cite

View full text Add to dashboard Cite

Abstract. Large-scale data analysis has increasingly come to rely on MapReduce and its open-source implementation Hadoop. Recently, Hadoop has not only been used for running single batch jobs but it has also been optimized to simultaneously support the execution of multiple jobs belonging to multiple concurrent users. Several schedulers (i.e., Fifo, Fair, and Capacity schedulers) have been proposed to optimize locality executions of tasks but do not consider failures, although, evidence in the literature shows that faults do occur and can probably result in performance problems. In this paper, we have designed a set of experiments to evaluate the performance of Hadoop under failure when applying several schedulers (i.e., explore the conflict between job scheduling, exposing locality executions, and failures). Our results reveal several drawbacks of current Hadoop's mechanism in prioritizing failed tasks. By trying to launch failed tasks as soon as possible regardless of locality, it significantly increases the execution time of jobs with failed tasks, due to two reasons: 1) available resources might not be freed up as quickly as expected and 2) failed tasks might be re-executed on machines with no data on it, introducing extra cost for data transferring through network, which is normally the most scarce resource in today's data-centers. Our preliminary study with Hadoop not only helps us to understand the interplay between fault-tolerance and job scheduling, but also offers useful insights into optimizing the current schedulers to be more efficient in case of failures.

show abstract

Section: Discussion On Hadoop Performance Under Failurementioning

confidence: 99%

An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop’s Schedulers Under Failures

Ibrahim

Phuong

Antoniu

2015

Adaptive Resource Management and Scheduling for Cloud Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…The Maestro is a replica aware scheduler algorithm introduced by S. Ibrahim, H. Jin, et al This technique defers the difficulty of non-local tasks that depend on replica of map tasks [43]. The Maestro keeps track of the chunks and duplicate locations, along with the number of other chunks hosted by each node.…”

Section: Maestro Scheduling Algorithmmentioning

confidence: 99%

The Ambient Scrutinize of Scheduling Algorithms in Big Data Territory.

Perwej¹

2018

IJAR

View full text Add to dashboard Cite

“…Replication has become an essential feature in storage systems and is extensively leveraged in cloud environments [3][4] [5]. It is the main reason behind several features such as fast accesses, enhanced performance, and high availability.…”

Section: Figure 1 Leveraging Geographically-distributed Data Replicamentioning

confidence: 99%

Consistency in the Cloud: When Money Does Matter!

Chihoub

Ibrahim²,

Antoniu³

et al. 2013

2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing

View full text Add to dashboard Cite

Abstract-With the emergence of cloud computing, many organizations have moved their data to the cloud in order to provide scalable, reliable and highly available services. To meet ever growing user needs, these services mainly rely on geographically-distributed data replication to guarantee good performance and high availability. However, with replication, consistency comes into question.Service providers in the cloud have the freedom to select the level of consistency according to the access patterns exhibited by the applications. Most optimizations efforts then concentrate on how to provide adequate trade-offs between consistency guarantees and performance. However, as the monetary cost completely relies on the service providers, in this paper we argue that monetary cost should be taken into consideration when evaluating or selecting a consistency level in the cloud. Accordingly, we define a new metric called consistency-cost efficiency. Based on this metric, we present a simple, yet efficient economical consistency model, called Bismar, that adaptively tunes the consistency level at run-time in order to reduce the monetary cost while simultaneously maintaining a low fraction of stale reads. Experimental evaluations with the Cassandra cloud storage on a Grid'5000 testbed show the validity of the metric and demonstrate the effectiveness of the proposed consistency model.

show abstract

Maestro: Replica-Aware Map Scheduling for MapReduce

Cited by 87 publications

References 15 publications

An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop’s Schedulers Under Failures

An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop’s Schedulers Under Failures

The Ambient Scrutinize of Scheduling Algorithms in Big Data Territory.

Consistency in the Cloud: When Money Does Matter!

Contact Info

Product

Resources

About