Hybrid HDFS: decreasing energy consumption and speeding up Hadoop using SSDs

Abstract

Apache Hadoop has evolved significantly over the last years, with more than 60 releases bringing new features. By implementing the MapReduce programming paradigm and leveraging HDFS, its distributed file system, Hadoop has become a reliable and fault tolerant middleware for parallel and distributed computing over large datasets.Nevertheless, Hadoop may struggle under certain workloads, resulting in poor performance and high energy consumption. Users increasingly demand that high performance computing solutions being to address sustainability and limit power consumption. In this paper, we introduce HDFS H , a hybrid storage mechanism for HDFS, which uses a combination of Hard Disks and Solid-State Disks to achieve higher performance while saving power in Hadoop computations. HDFS H brings to middleware the best from HDs (affordable cost per GB and high storage capacity) and SSDs (high throughput and low energy consumption) in a configurable fashion, using dedicated storage zones for each storage device type. We implemented our mechanism as a block placement policy for HDFS, and assessed it over six recent releases of the Hadoop project, representing different designs of the Hadoop middleware. Results indicate that our approach increases overall job performance while decreasing the energy consumption under most hybrid configurations evaluated. Our results also showed that in many cases storing only part of the data in SSDs results in significant energy savings and execution speedups. ABSTRACTApache Hadoop has evolved significantly over the last years, with more than 60 releases bringing new features. By implementing the MapReduce programming paradigm and leveraging HDFS, its distributed file system, Hadoop has become a reliable and fault tolerant middleware for parallel and distributed computing over large datasets. Nevertheless, Hadoop may struggle under certain workloads, resulting in poor performance and high energy consumption. Users increasingly demand that high performance computing solutions being to address sustainability and limit power consumption. In this paper, we introduce HDFSH , a hybrid storage mechanism for HDFS, which uses a combination of Hard Disks and Solid-State Disks to achieve higher performance while saving power in Hadoop computations. HDFSH brings to middleware the best from HDs (affordable cost per GB and high storage capacity) and SSDs (high throughput and low energy consumption) in a configurable fashion, using dedicated storage zones for each storage device type. We implemented our mechanism as a block placement policy for HDFS, and assessed it over six recent releases of the Hadoop project, representing different designs of the Hadoop middleware. Results indicate that our approach increases overall job performance while decreasing the energy consumption under most hybrid configurations evaluated. Our results also showed that in many cases storing only part of the data in SSDs results in significant energy savings and execution speedups.

Evaluating the Impact of Design Pattern Usage on Energy Consumption of Applications for Mobile Platform

“…The authors of [28] have told us that a number of users nowadays increasingly demand high-performance computing solutions being able to address sustainability and limit power consumption. They have also given an HDFS approach, a hybrid storage mechanism, which uses hard disk and solid-state disk combination to get better performance and save energy.…”

Transfer Learning Based Performance Modeling And Effective Storage Management In Big Data Ecosystems

“…Adding SSDs into the storage system is one of the most efficient ways to improve I/O intensive workload performance and reduce energy consumption in big data ecosystems [56]. Usually, SSD drives outperform HDD drives due to the SSD and HDD hardware architecture design differentiation [13].…”

“…reduce cost [60] [62] and decrease energy consumption [56]. However, SSD is not always the optimal storage for MapReduce workloads.…”

“…Controlling SSD from a Hadoop level provides advantages for both the workload and SSD. It is widely recognize that SSD brings huge benefits into the Hadoop ecosystem [59], [61].The benefits include improved I/O [103], decreased energy consumption [56] and reduced overall cost [60] [62]. It is also evident that SSD is expensive and has limited capacity.…”