IO dependent SSD cache allocation for elastic Hadoop applications

Abstract

Elastic Hadoop applications consisting of multiple virtual machines (VMs) are widely used to support big data analysis and processing. In this scenario, flash-based solid state drive (SSD) is usually deployed on hypervisors and used as the cache to improve the IO performance. However, existing SSD caching schemes are mostly VM-centric, which focus on the low-level IO performance metrics of individual VMs. They may not lead to the optimized performance of elastic Hadoop applications, i.e., the job completion time (JCT), as the importance of VMs inside the application are different even though they have the similar low-level IO patterns. Considering the IO dependency among VMs and figuring out the importance, which we regard as the application-centric metrics, may potentially better improve the performance. We present IO dependency based requirement model, to characterize the requirement of SSD cache for each VM inside the elastic Hadoop application, and then use it in a genetic algorithm (GA) based approach to calculate the nearly optimal weights of VMs for allocating the per-VM SSD cache space and the capacity of the I/O operations per second (IOPS). Furthermore, we present a tool AC-SSD based on the approach and introduce the closed-loop adaptation to react to continuously changing workloads. The evaluation shows that by using AC-SSD, the JCT is reduced by up to 39% for IO sensitive workloads, up to 29% for continuously changing workloads, and over 12.5% for different scale of data comparing to the shared cache.

Transfer Learning Based Performance Modeling And Effective Storage Management In Big Data Ecosystems

“…For example, MapReduce based ecosystems such as Hadoop execute workloads on two stages, Map and Reduce. Each of them interacts with the local and the remote storage differently [8] .…”

“…Previous research have shown that utilizing SSD drives can improve big data workloads' performance but improvement depending on the workload differently based on the workload type and SSD usage approach. In the Hadoop ecosystem, several studies investigated the performance improvement of a variety of MapReduce workloads when introducing SSD into the ecosystem as a tier [59], [60], [61] [12] or as a cache [8]. To examine the difference in SSD utilization usage approaches impact on MapReduce workloads, [62] and [61] compared the performance improvement when applying SSD as a tier and as a cache.…”

“…In a big data ecosystem, the workload has two types of data: intermediate and final. Although intermediate data seems to utilize storage lighter than the final result, studies show that some workloads have heavy intermediate data [8].…”

“…Managing SSD's lifespan at the hardware level allows these policies to control what the storage receives, but software level management can define policies to control what sent into the storage in first place. There is relatively a small body of literature that is concerned with managing SSD from a software perspective such as [10] which defines I/O placement policies into SSDs based on SSDs cost, and [8] that define SSD cache allocation in the Hadoop ecosystem. However, none of them consider SSD's lifespan consumption and SSD's usage approaches which we cover in this thesis.…”

An Application-Oriented Cache Allocation and Prefetching Method for Long-Running Applications in Distributed Storage Systems

“…In order to improve cache performance, numerous efforts on cache allocation are proposed for parallel computing [16] , chip [7] , virtual machines [17,18] and Web search engines [19] . Venkatesan et al [18] figure out the miss ratio curve of virtual machines are varied and propose a dynamic approach to partition cleancache fairly.…”