A Study of Big Data Computing Platforms: Fairness and Energy Consumption

Niu, Zhaojie; He, Bingsheng

doi:10.1109/ic2ew.2016.31

Cited by 4 publications

(4 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We conjecture the concept of joule efficiency can also be applied to other applications. GreenMR [40], [39] and GreenPar [20] consider joule efficiency when leveraging green energy. However, they do not study impact of on-peak/offpeak pricing on the cost in MapReduce/Hadoop cluster.…”

Section: Related Workmentioning

confidence: 99%

JouleMR: Towards Cost-Effective and Green-Aware Data Processing Frameworks

Niu

Liu

2018

IEEE Trans. Big Data

Self Cite

View full text Add to dashboard Cite

Interests have been growing in energy management of the cluster effectively in order to reduce the energy consumption as well as the electricity cost. Renewable energy and dynamic pricing schemes in smart grids are two major emerging trends in energy markets. However, current data processing frameworks are not aware of the efficiency of each joule consumed by the data center workloads in the context of these two major trends. In fact, not all joules are equal in the sense that the amount of work that can be done by a joule can vary significantly in data centers. Ignoring this fact leads to significant energy waste (by 25% of the total energy consumption in Hadoop YARN on a Facebook production trace according to our study). In this paper, we propose JouleMR, a cost-effective and green-aware data processing framework. Specifically, we investigate how to exploit such joule efficiency to maximize the benefits of renewable energy as well as dynamic pricing schemes for MapReduce framework. We develop job/task scheduling algorithms with a particular focus on the factors on joule efficiency in the data center, including the energy efficiency of MapReduce workloads, renewable energy supply, dynamic pricing and the battery usage. We further develop a simple yet effective performance-energy consumption model to guide our scheduling decisions. We have implemented JouleMR on top of Hadoop YARN. The experiments demonstrate the accuracy of our models, and the effectiveness of our cost-effective and green-aware optimizations outperform the state-of-the-art implementations over Hadoop YARN.

show abstract

Section: Related Workmentioning

confidence: 99%

JouleMR: Towards Cost-Effective and Green-Aware Data Processing Frameworks

Niu

Liu

2018

IEEE Trans. Big Data

Self Cite

View full text Add to dashboard Cite

show abstract

“…19. This tradeoff also depends on the workload that further verifies the importance of the adaptive scheduler in bi-criteria optimization for resource efficiency and fairness.…”

mentioning

confidence: 85%

“…However, they do not consider the tradeoff between the resource efficiency, job latency, fairness and energy consumption. Therefore, new schedulers [16,17,18,19] are proposed to address the tradeoff between these discordant factors.…”

Section: Hadoop Yarnmentioning

confidence: 99%

See 1 more Smart Citation

A study of big data computing platforms : performance, fairness and energy consumption

Niu¹

Self Cite

View full text Add to dashboard Cite

Today, population aging is a trend spreading across the world. It is estimated that the number of people aged 60 years or over, is increasing rapidly. Although the elderly has wisdom and wealth gathered from their life experience, they often require long-term assistance from others. Ambient assisted living (AAL) systems open up a new opportunity to address the needs of aged by utilizing information and communication techniques. Multiple sensors and actuators coexist in ambient assisted living systems and are deployed everywhere surrounding elderly. A huge volume of sensed data is collected continuously from elderly, storage of these massive raw data and efficient processing on them to infer knowledge from the data and also guide the actuators to meet the needs of elderly is a problem we face. In the thesis, we mainly focus on the efficient large-scale data processing, we investigate whether and how we can build an effective large-scale data processing system to effectively excavate knowledge, wisdom and skills from elderly people to impact the development of the entire society.A lot of large-scale big data analytics systems have thereby emerged to process the massive data effectively. Efficient job scheduling and resource management for these data analytic frameworks are nontrivial. Modern job schedulers and resource coordinators in data processing frameworks often need to consider multiple objectives simultaneously due to various system operators requirements on data analytics. Currently, resource efficiency (throughput), job latency (per-job performance), fairness (isolation guarantee) and energy consumption are important concerns for the job scheduler in modern large-scale multiple-tenant environments. Resource efficiency is de facto the very important factor for the big data analytic framework. Job latency reflects the waiting time of the application. Fairness is a key building block of any multiple-tenant computing system that allows resource sharing effectively. Energy usage of the data center has reached 3% of the global electricity consumption while generating 200 million metric tons of CO2 in 2014. In order to reduce carbon emission and financial burden on the electricity, a lot of data centers have been re-designed and powered with multiple energy sources, including renewable (green) energy from non-polluting sources and brown energy from traditional polluting sources.Improving the resource efficiency and reducing the per-job latency are the common sense on those large-scale data processing frameworks and fruitful studies are proposed in these directions. In i I would like to thank my friends and colleagues in IGS-LILY and PDSL. I will always remember all the great memories we created together in the part four years.I gratefully acknowledge the scholarship provided by LILY, IGS(Interdisciplinary Graduate School) and Nanyang Technological University for my doctorial program.Last but not least, I want to thank my family and friends for their unwavering support throughput my PhD.

show abstract