JouleMR: Towards Cost-Effective and Green-Aware Data Processing Frameworks

Abstract

Interests have been growing in energy management of the cluster effectively in order to reduce the energy consumption as well as the electricity cost. Renewable energy and dynamic pricing schemes in smart grids are two major emerging trends in energy markets. However, current data processing frameworks are not aware of the efficiency of each joule consumed by the data center workloads in the context of these two major trends. In fact, not all joules are equal in the sense that the amount of work that can be done by a joule can vary significantly in data centers. Ignoring this fact leads to significant energy waste (by 25% of the total energy consumption in Hadoop YARN on a Facebook production trace according to our study). In this paper, we propose JouleMR, a cost-effective and green-aware data processing framework. Specifically, we investigate how to exploit such joule efficiency to maximize the benefits of renewable energy as well as dynamic pricing schemes for MapReduce framework. We develop job/task scheduling algorithms with a particular focus on the factors on joule efficiency in the data center, including the energy efficiency of MapReduce workloads, renewable energy supply, dynamic pricing and the battery usage. We further develop a simple yet effective performance-energy consumption model to guide our scheduling decisions. We have implemented JouleMR on top of Hadoop YARN. The experiments demonstrate the accuracy of our models, and the effectiveness of our cost-effective and green-aware optimizations outperform the state-of-the-art implementations over Hadoop YARN.

Big data analytics in smart grids: state‐of‐the‐art, challenges, opportunities, and future directions

“…Hadoop is an open source framework for storing and processing large datasets using the MapReduce programming model [143]. The Hadoop consists of a storage part (known as Hadoop distributed file systems (HDFS)) and processing part (known as MapReduce programming model) [138][139][140]. Primarily, Hadoop splits files into large blocks and distributes them across nodes so as to process data in parallel.…”

Wireless Powered Mobile Edge Computing for Industrial Internet of Things Systems

“…Edge nodes can harvest green energy like solar from the ambient environment as a supplementary for grid power. Although highly variable and unpredictable, the green energy can still be directly employed for some delay-tolerant services [27] or be stored in EHSM before it accumulates to a certain level for use. Different from edge nodes with relatively complicated facilities to collect energy, low-cost devices such as sensors and actuators depend on edge nodes to continuously recharge their EHSMs through WPT to prolong lifetime.…”