MARLA: MapReduce for Heterogeneous Clusters

Fadika, Zacharia; Dede, Elif; Hartog, Jessica; Govindaraju, Madhusudhan

doi:10.1109/ccgrid.2012.135

Cited by 36 publications

(17 citation statements)

References 13 publications

(20 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recall from the previous section that this same scenario also changes the power consumption of an individual node. Note that if our framework is properly tuned to have the appropriate boundary temperature for the given cluster, we can realize turnaround times on par with those discussed in [14]. Since we know that our execution times are similar to those of MARLA, we can see that power savings on the cluster level may be achieved.…”

Section: Comparison With Marlamentioning

confidence: 51%

“…The primary difference between our framework and MARLA is that we have scheduled this framework for energy awareness, whereas MARLA schedules for performance in heterogeneous clusters alone. Comparing our framework to MARLA is sufficient as MARLA has been compared to Hadoop and Mariane in our previous work [14]; showing improved performance in heterogeneous clusters.…”

Section: Comparison With Marlamentioning

confidence: 99%

See 1 more Smart Citation

Configuring a MapReduce Framework for Dynamic and Efficient Energy Adaptation

Hartog

Fadika

Dede

et al. 2012

2012 IEEE Fifth International Conference on Cloud Computing

Self Cite

View full text Add to dashboard Cite

Abstract-MapReduce has become a popular framework for Big Data applications. While MapReduce has received much praise for its scalability and efficiency, it has not been thoroughly evaluated for power consumption. Our goal with this paper is to explore the possibility of scheduling in a power-efficient manner without the need for expensive power monitors on every node. We begin by considering that no cluster is truly homogeneous with respect to energy consumption. From there we develop a MapReduce framework that can evaluate the current status of each node and dynamically react to estimated power usage. In so doing, we shift power consumption work toward more energy efficient nodes which are currently consuming less power. Our work shows that given an ideal framework configuration, certain nodes may consume only 62.3% of the dynamic power they consumed when the same framework was configured as it would be in a traditional MapReduce implementation.

show abstract

Section: Comparison With Marlamentioning

confidence: 51%

Section: Comparison With Marlamentioning

confidence: 99%

Configuring a MapReduce Framework for Dynamic and Efficient Energy Adaptation

Hartog

Fadika

Dede

et al. 2012

2012 IEEE Fifth International Conference on Cloud Computing

Self Cite

View full text Add to dashboard Cite

show abstract

“…In [15] a MapReduce framework for heterogeneous and loadimbalanced environments is described. The research presented in [16] and [17] focuses on a formulation of the MapReduce matchmaking and scheduling problem using linear programming.…”

Section: Related Workmentioning

confidence: 99%

A Constraint Programming Based Hadoop Scheduler for Handling MapReduce Jobs with Deadlines on Clouds

Lim

Majumdar

Ashwood-Smith

2015

Proceedings of the 6th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

A novel MapReduce constraint programming based matchmaking and scheduling algorithm (MRCP) that can handle MapReduce jobs with deadlines and achieve high system performance is devised. The MRCP algorithm is incorporated into Hadoop, which is a widely used open source implementation of the MapReduce programming model, as a new scheduler called the CP-Scheduler. This paper originates from the collaborative research with our industrial partner concerning the engineering of resource management middleware for high performance. It describes our experiences and the challenges that we encountered in designing and implementing the prototype CP-based Hadoop scheduler. A detailed performance evaluation of the CP-Scheduler is conducted on Amazon EC2 to determine the CP-Scheduler's effectiveness as well as to obtain insights into system behaviour and performance. In addition, the CP-Scheduler's performance is also compared with an earliest deadline first (EDF) Hadoop scheduler, which is implemented by extending Hadoop's default FIFO scheduler. The experimental results demonstrate the effectiveness of the CPScheduler's ability to handle an open stream of MapReduce jobs with deadlines in a Hadoop cluster.

show abstract

“…This performance divergence between the two frameworks can be explained by the fact that Hadoop start-up cost is more visible in small data sizes. [10], [12], [13]. The MARISSA framework is designed as a lightweight MapReduce platform where the start-up overhead is minimal.…”

Section: B Data Transformation (Mr1)mentioning

confidence: 99%

A Processing Pipeline for Cassandra Datasets Based on Hadoop Streaming

Dede

Sendir

Kuzlu

et al. 2014

2014 IEEE International Congress on Big Data

View full text Add to dashboard Cite

Abstract- 1The progressive transition in the nature of both scientific and industrial datasets has been the driving force behind the development and research interests in the NoSQL data model. Loosely structured data poses a challenge to traditional data store systems, and when working with the NoSQL model, these systems are often considered impractical and expensive. As the quantity of unstructured data grows, so does the demand for a processing pipeline that is capable of seamlessly combining the NoSQL storage model and a "Big Data" processing platform such as MapReduce. Although, MapReduce is the paradigm of choice for data-intensive computing, Java-based frameworks such as Hadoop require users to write MapReduce code in Java. Hadoop Streaming, on the other hand, allows users to define non-Java executables as map and reduce operations. Similarly, for legacy C/C++ applications and other non-Java executables, there is a need to allow NoSQL data stores access to the features of Hadoop Streaming. In this paper, we present approaches in solving the challenge of integrating NoSQL data stores with MapReduce for non-Java application scenarios, along with advantages and disadvantages of each approach. We compare Hadoop Streaming alongside our own streaming framework, MARISSA, to show performance implications of coupling NoSQL data stores like Cassandra with MapReduce frameworks that normally rely on file-system based data stores.

show abstract

MARLA: MapReduce for Heterogeneous Clusters

Cited by 36 publications

References 13 publications

Configuring a MapReduce Framework for Dynamic and Efficient Energy Adaptation

Configuring a MapReduce Framework for Dynamic and Efficient Energy Adaptation

A Constraint Programming Based Hadoop Scheduler for Handling MapReduce Jobs with Deadlines on Clouds

A Processing Pipeline for Cassandra Datasets Based on Hadoop Streaming

Contact Info

Product

Resources

About