Cost-Effective Resource Provisioning for MapReduce in a Cloud

Abstract

Abstract-This paper presents a new MapReduce cloud service model, Cura, for provisioning cost-effective MapReduce services in a cloud. In contrast to existing MapReduce cloud services such as a generic compute cloud or a dedicated MapReduce cloud, Cura has a number of unique benefits. Firstly, Cura is designed to provide a cost-effective solution to efficiently handle MapReduce production workloads that have a significant amount of interactive jobs. Secondly, unlike existing services that require customers to decide the resources to be used for the jobs, Cura leverages MapReduce profiling to automatically create the best cluster configuration for the jobs. While the existing models allow only a per-job resource optimization for the jobs, Cura implements a globally efficient resource allocation scheme that significantly reduces the resource usage cost in the cloud. Thirdly, Cura leverages unique optimization opportunities when dealing with workloads that can withstand some slack. By effectively multiplexing the available cloud resources among the jobs based on the job requirements, Cura achieves significantly lower resource usage costs for the jobs. Cura's core resource management schemes include cost-aware resource provisioning, VM-aware scheduling and online virtual machine reconfiguration. Our experimental results using Facebook-like workload traces show that our techniques lead to more than 80% reduction in the cloud compute infrastructure cost with upto 65% reduction in job response times.

Optimized Speculative Execution to Improve Performance of MapReduce Jobs on Virtualized Computing Environment

“…Reference [25] introduces a Quincy scheduler to achieve data locality. Several recent proposals, such as resource-aware adaptive scheduling [26] and cost effective resource provisioning [27], have introduced resource-aware job schedulers to the MapReduce framework. Reference [28] mentions the problem of task assignment with the consideration of the data locality in cloud computing.…”

An adaptive approach to better load balancing in a consumer-centric cloud environment

“…[11] proposed a method to allocate more data to a node that has better performance. By monitoring running map tasks and reduce tasks, Hadoop ensures that all tasks running on each node can substantially complete.…”

“…Therefore, optimization schemes have been proposed [10][11][12][13][14][15][16][17][18][19], but most of them are only focused on task execution time, whereas storage space is often neglected.…”

Resource Scheduling in Cloud Environmet: A Survey

“…• to provide a cost-effective solution to efficiently handle MapReduce production workloads • leverages MapReduce profiling to automatically create the best cluster configuration for the jobs • implements a globally efficient resource allocation scheme Cost, Execution time Java based simulator The experimental results using Facebook-like workload traces show that proposed techniques lead to more than 80% reduction in the cloud compute infrastructure cost with up to 65% reduction in job response times [25] A dynamic virtual resource renting method that attempts to dynamically adjust the virtual resource rental strategy according to price distribution and task urgency Cost Java based simulator The simulation results show that average rental cost of proposed method is much lower and average profit is the highest among other traditional revenue-aware algorithms [26] …”