VENU: Orchestrating SSDs in hadoop storage

Krish, K. R.; Iqbal, Murium; Butt, Ali R.

doi:10.1109/bigdata.2014.7004234

Cited by 21 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Apache Hadoop community [36] has also proposed such kind of centralized cache management scheme in HDFS, which is an explicit caching mechanism that allows users to specify paths to be cached by HDFS. Some recent studies [20,25,26,35] also pay attention to incorporate heterogeneous storage media (e.g. SSD and parallel file system) in HDFS.…”

Section: Related Workmentioning

confidence: 99%

“…SSD and parallel file system) in HDFS. Authors in [25] deal with data distribution in the presence of nodes that do not have uniform storage characteristics; whereas, [26] caches data in SSD. Researchers in [35] present HDFS-specific optimizations for PVFS and [32] propose to store cold data of Hadoop cluster to network-attached file systems.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

High Performance Design for HDFS with Byte-Addressability of NVM and RDMA

Islam

Wasi-ur-Rahman

et al. 2016

Proceedings of the 2016 International Conference on Supercomputing

View full text Add to dashboard Cite

Non-Volatile Memory (NVM) offers byte-addressability with DRAM like performance along with persistence. Thus, NVMs provide the opportunity to build high-throughput storage systems for data-intensive applications. HDFS (Hadoop Distributed File System) is the primary storage engine for MapReduce, Spark, and HBase. Even though HDFS was initially designed for commodity hardware, it is increasingly being used on HPC (High Performance Computing) clusters. The outstanding performance requirements of HPC systems make the I/O bottlenecks of HDFS a critical issue to rethink its storage architecture over NVMs. In this paper, we present a novel design for HDFS to leverage the byteaddressability of NVM for RDMA (Remote Direct Memory Access)-based communication. We analyze the performance potential of using NVM for HDFS and re-design HDFS I/O with memory semantics to exploit the byte-addressability fully. We call this design NVFS (NVM-and RDMA-aware HDFS). We also present cost-effective acceleration techniques for HBase and Spark to utilize the NVM-based design of HDFS by storing only the HBase Write Ahead Logs and Spark job outputs to NVM, respectively. We also propose enhancements to use the NVFS design as a burst buffer for running Spark jobs on top of parallel file systems like Lustre. Performance evaluations show that our design can improve the write and read throughputs of HDFS by up to 4x and 2x, respectively. The execution times of data generation benchmarks are reduced by up to 45%. The proposed design also reduces the overall execution time of the SWIM workload by up to 18% over HDFS with a maximum benefit of 37% for job-38. For Spark TeraSort, our proposed scheme yields a performance gain of up to 11%. The performances of HBase insert, update, and read operations are improved by 21%, 16%, and 26%, respectively. Our NVM-based burst buffer can improve the I/O performance of Spark PageRank by up *

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

High Performance Design for HDFS with Byte-Addressability of NVM and RDMA

Islam

Wasi-ur-Rahman

et al. 2016

Proceedings of the 2016 International Conference on Supercomputing

View full text Add to dashboard Cite

show abstract

“…In the Hadoop ecosystem, several studies investigated the performance improvement of a variety of MapReduce workloads when introducing SSD into the ecosystem as a tier [59], [60], [61] [12] or as a cache [8]. To examine the difference in SSD utilization usage approaches impact on MapReduce workloads, [62] and [61] compared the performance improvement when applying SSD as a tier and as a cache.…”

Section: Storage Capability and Workload Characteristicmentioning

confidence: 99%

Transfer Learning Based Performance Modeling And Effective Storage Management In Big Data Ecosystems

Alsayoud¹

2021

Preprint

View full text Add to dashboard Cite

Big data ecosystems contain a mix of sophisticated hardware storage components to support heterogeneous workloads. Storage components and the workloads interact and affect each other; therefore, their relationship has to consider when modeling workloads or managing storage. Efficient workload modeling guides optimal storage management decisions, and the right decisions help guarantee the workload’s needs. The first part of this thesis focuses on workload modeling efficiency, and the second part focuses on cost-effective storage management.<div>Workload performance modeling is an essential step in management decisions. The standard modeling approach constructs the model based on a historical dataset collected from one set of setups (scenario). The standard modeling approach requires the model to be reconstructed from scratch with every time the setups changes. To address this issue, we propose a cross-scenario modeling approach that improves the workload’s performance classification accuracy by up to 78% through adopting the Transfer Learning (TL).<br></div><div>The storage system is the most crucial component of the big data ecosystem, where the workload’s execution process starts by fetching data from it and ends by storing data into it. Thus, the workload’s performance is directly affected by storage capability. To provide a high I/O performance in the ecosystems, Solid State Drive (SSD) are utilized as a tier or as a cache on big data distributed ecosystems. SSDs have a short lifespan that is affected by data size and the number of writing operations. Balancing performance requirements and SSD’s lifespan consumption is never easy, and it’s even harder when interacting with a huge amount of data and with heterogeneous I/O patterns. In this thesis, we analysis big data workloads I/O pattern impacts on SSD’s lifespan when SSD is used as a tier or as a cache. Then, we design a Hidden Markov Model (HMM) based I/O pattern controller that manages workload placement and guarantees cost-effective storage that enhances the workload performance by up to 60%, and improves SSD’s lifespan by up to 40%. </div><div>The designed transfer learning modeling approach and the storage management solutions improve workload modeling accuracy, and the quality of the storage management policies while the testing setup changes.<br></div>

show abstract

“…[18] explore the benefits and limitations of in-storage processing on SSDs (the execution of applications on processors in the storage controller). Other researchers focus on incorporating SSDs into HDFS using caching mechanisms to achieve better performance [20,28,32,35]. A few works also discuss SSDs' impact on Hadoop [17,25], sometimes focusing on using SSDs as the sole storage device under HDFS.…”

Section: Related Workmentioning

confidence: 99%

Hybrid HDFS: decreasing energy consumption and speeding up Hadoop using SSDs

Polato

Barbosa

Hindle

et al. 2015

Preprint

View full text Add to dashboard Cite

Apache Hadoop has evolved significantly over the last years, with more than 60 releases bringing new features. By implementing the MapReduce programming paradigm and leveraging HDFS, its distributed file system, Hadoop has become a reliable and fault tolerant middleware for parallel and distributed computing over large datasets.Nevertheless, Hadoop may struggle under certain workloads, resulting in poor performance and high energy consumption. Users increasingly demand that high performance computing solutions being to address sustainability and limit power consumption. In this paper, we introduce HDFS H , a hybrid storage mechanism for HDFS, which uses a combination of Hard Disks and Solid-State Disks to achieve higher performance while saving power in Hadoop computations. HDFS H brings to middleware the best from HDs (affordable cost per GB and high storage capacity) and SSDs (high throughput and low energy consumption) in a configurable fashion, using dedicated storage zones for each storage device type. We implemented our mechanism as a block placement policy for HDFS, and assessed it over six recent releases of the Hadoop project, representing different designs of the Hadoop middleware. Results indicate that our approach increases overall job performance while decreasing the energy consumption under most hybrid configurations evaluated. Our results also showed that in many cases storing only part of the data in SSDs results in significant energy savings and execution speedups. ABSTRACTApache Hadoop has evolved significantly over the last years, with more than 60 releases bringing new features. By implementing the MapReduce programming paradigm and leveraging HDFS, its distributed file system, Hadoop has become a reliable and fault tolerant middleware for parallel and distributed computing over large datasets. Nevertheless, Hadoop may struggle under certain workloads, resulting in poor performance and high energy consumption. Users increasingly demand that high performance computing solutions being to address sustainability and limit power consumption. In this paper, we introduce HDFSH , a hybrid storage mechanism for HDFS, which uses a combination of Hard Disks and Solid-State Disks to achieve higher performance while saving power in Hadoop computations. HDFSH brings to middleware the best from HDs (affordable cost per GB and high storage capacity) and SSDs (high throughput and low energy consumption) in a configurable fashion, using dedicated storage zones for each storage device type. We implemented our mechanism as a block placement policy for HDFS, and assessed it over six recent releases of the Hadoop project, representing different designs of the Hadoop middleware. Results indicate that our approach increases overall job performance while decreasing the energy consumption under most hybrid configurations evaluated. Our results also showed that in many cases storing only part of the data in SSDs results in significant energy savings and execution speedups.

show abstract

VENU: Orchestrating SSDs in hadoop storage

Cited by 21 publications

References 10 publications

High Performance Design for HDFS with Byte-Addressability of NVM and RDMA

High Performance Design for HDFS with Byte-Addressability of NVM and RDMA

Transfer Learning Based Performance Modeling And Effective Storage Management In Big Data Ecosystems

Hybrid HDFS: decreasing energy consumption and speeding up Hadoop using SSDs

Contact Info

Product

Resources

About