DELMA: Dynamically ELastic MapReduce Framework for CPU-Intensive Applications

Abstract

Abstract-Since its introduction, MapReduce implementations have been primarily focused towards static compute cluster sizes. In this paper, we introduce the concept of dynamic elasticity to MapReduce. We present the design decisions and implementation tradeoffs for DELMA, (Dynamically ELastic MApReduce), a framework that follows the MapReduce paradigm, just like Hadoop MapReduce, but that is capable of growing and shrinking its cluster size, as jobs are underway. In our study, we test DELMA in diverse performance scenarios, ranging from diverse node additions to node additions at various points in the application run-time with various dataset sizes. The applicability of the MapReduce paradigm extends far beyond its use with large-scale data intensive applications, and can also be brought to bear in processing long running distributed applications executing on small-sized clusters. In this work, we focus both on the performance of processing hierarchical data in distributed scientific applications, as well as the processing of smaller but demanding input sizes primarily used in small clusters. We run experiments for datasets that require CPU intensive processing, ranging in size from millions of input data elements to process, up to over half a billion elements, and observe the positive scalability patterns exhibited by the system. We show that for such sizes, performance increases with data and cluster size increases. We conclude on the benefits of providing MapReduce with the capability of dynamically growing and shrinking its cluster configuration by adding and removing nodes during jobs, and explain the possibilities presented by this model.

A Queuing Model to Achieve Proper Elasticity for Cloud Cluster Jobs

“…Even with these integrated solutions, determining the number of slave nodes for mappers and reducers is almost done arbitrarily and is in fact a trial and error exercise that will yield to multiple SLO violations in the course of adjusting the number of mappers and reducers. Fadika & Govindaraju (2011) proposed and implemented a dynamically elastic MapReduce framework in which the number of slave nodes is dynamically adjusted during the job execution. Authors had shown such a scheme can boost the performance of job execution by 50%.…”

MARIANE: MApReduce Implementation Adapted for HPC Environments

“…EC2 and Azure are proprietary applications, and hence an insightful analysis on their design decisions is not possible. Twister [16] and DELMA [29] are MapReduce frameworks, one espousing an iterative approach, and the other an elastic approach to solving MapReduce problems. Both however require each of their nodes to benefit from individual storage units, as a shared storage option is not yet supported by both implementations.…”

Towards QoS-Oriented SLA Guarantees for Online Cloud Services

“…It specifies that the MapReduce job response time should In order to guarantee the SLA, we applied a controltheoretic approach to provide a SLA-oriented self-elastic MapReduce cluster. Although some initiatives exist to add elasticity to MapReduce [14], [15], as far as we know, this is the first attempt to provide fully self-elastic MapReduce that is able to automatically adapt cluster size to workload variations in order to guarantee the SLA. To this purpose, the SLA is translated into a utility function in an ad hoc manner.…”