Performance and energy efficiency of big data applications in cloud environments: A Hadoop case study

Abstract

The exponential growth of scientific and business data has resulted in the evolution of the cloud computing environments and the MapReduce parallel programming model. The focus of cloud computing is increased utilization and power savings through consolidation while MapReduce enables large scale data analysis.Hadoop, an open source implementation of MapReduce has gained popularity in the last few years. In this paper, we evaluate Hadoop performance in both the traditional model of collocated data and compute services as well as consider the impact of separating out the services. The separation of data and compute services provides more flexibility in environments where data locality might not have a considerable impact such as virtualized environments and clusters with advanced networks. In this paper, we also conduct an energy efficiency evaluation of Hadoop on physical and virtual clusters in different configurations. Our extensive evaluation shows that: (1) coexisting virtual machines on servers decrease the disk throughput; (2) performance on physical clusters is significantly better than on virtual clusters; (3) performance degradation due to separation of the services depends on the data to compute ratio; (4) application completion progress correlates with the power consumption and power consumption is heavily application specific. Finally, we present a discussion on the implications of using cloud environments for big data analyses.

Adaptation in cloud resource configuration: a survey

“…The authors in [31] use a load predictor that clusters historical resource utilisation, and select the cluster set with the highest similarity as a training sample into a Neural Network. However, these approaches simplify the impact of colocating VMs, which can lead to significant performance overhead [83,84]. The authors in [85] tackle colocation interference and perform four parallel classifications on each application to evaluate the impact of vertical and horizontal scale, server configuration, and the impact of colocating applications.…”

Energy-Aware Task Scheduling Using Hybrid Firefly-BAT (FFABAT) in Big Data

“…The exodus of data and services provides more flexibility in surroundings, where data locality might not have a considerable impact. An energy efficiency assessment of Hadoop on physical and virtual clusters in different configurations is performed [31].…”

A large data processing algorithm for energy efficiency in a heterogeneous cluster

“…And research show that, in 2008, the world's 4400 servers consumed electricity 0.8% percent, if you go like that, at that rate By 2020, that proportion * Lei Wang: wl@ln.sgcc.com.cn will be 3.2%. Epa (US Environmental Protection Agency) issued a report statement in 2006, the total electricity consumption of American IT agencies was 61 billion KWh, the electricity bill alone is $4.5 billion [5]- [7]. So that's a concern Big data storage and processing performance must also be used for energy consumption Give enough attention [8].…”