Energy management for MapReduce clusters

Abstract

The area of cluster-level energy management has attracted significant research attention over the past few years. One class of techniques to reduce the energy consumption of clusters is to selectively power down nodes during periods of low utilization to increase energy efficiency. One can think of a number of ways of selectively powering down nodes, each with varying impact on the workload response time and overall energy consumption. Since the MapReduce framework is becoming "ubiquitous", the focus of this paper is on developing a framework for systematically considering various MapReduce node power down strategies, and their impact on the overall energy consumption and workload response time.We closely examine two extreme techniques that can be accommodated in this framework. The first is based on a recently proposed technique called "Covering Set" (CS) that keeps only a small fraction of the nodes powered up during periods of low utilization. At the other extreme is a technique that we propose in this paper, called the All-In Strategy (AIS). AIS uses all the nodes in the cluster to run a workload and then powers down the entire cluster. Using both actual evaluation and analytical modeling we bring out the differences between these two extreme techniques and show that AIS is often the right energy saving strategy.

Optimal Tradeoff between Energy Consumption and Response Time in Large-Scale MapReduce Clusters

“…However, there exist some studies that take into account both the energy consumption and the response time (e.g., [8], [9], [10]). In [8], two methods that target at the minimization of the energy consumption by activating or deactivating the nodes are presented. The main idea behind these methods is that, by deactivating some nodes, utilization and energy efficiency is improved thus leading to lower energy consumption.…”

“…The second method utilizes all the nodes in the cluster, and, upon the completion of task execution, powers down the nodes to avoid the high energy consumption at idle state. Several powered down states are examined in [8] from which hibernation seems the most efficient in terms of the trade-off between energy consumption and transition time (from online to offline state and vice versa). Our approach can be regarded as a hybrid of these two methods with the addition that we consider the case where high degrees of parallelism may not be beneficial.…”

“…Our approach can be regarded as a hybrid of these two methods with the addition that we consider the case where high degrees of parallelism may not be beneficial. Also, the energy consumption functions in our work are based on those in [8].…”

“…Finally, the value of k within this range that minimizes the energy consumption is computed and the algorithm finishes. The output of the algorithm defines the exact number of the machines to be used, whereas all the other nodes are powered down, e.g., to the hibernate state, which is characterized by low transition time and low energy consumption as explained in [8]. Because of the unimodality of the function, the tradeoff between energy and response time given a specific degradation threshold is optimal; any solution outside the range of values examined is inferior.…”

“…(3), it can be observed that the ratio between the power costs at different states is a key factor. According to [8], we can make the assumption that P tr is similar to the power P when the cluster is fully on and running a workload. However, T tr is defined by the capabilities of the hardware and the operating system; in general, we expect this to be orders of magnitude smaller that the total response time of intensive tasks.…”