MemEFS: An Elastic In-memory Runtime File System for eScience Applications

Abstract

Abstract-Data-intensive scientific workflows exhibit inter-task dependencies that generate file-based communication schemes. In such scenarios, traditional disk-based storage systems often limit overall application performance and scalability. To overcome the storage bottleneck, in-memory runtime distributed file systems speed up application I/O. Such systems are deployed statically onto a fixed number of compute nodes and act as a distributed, fast I/O cache for the runtime generated data. Such static deployment schemes have two major drawbacks. First, the user is faced with the sometimes difficult task of estimating the size of the generated data, as the application would fail otherwise. Second, because applications exhibit significant variability of the data footprint and of the achieved parallelism during their runtime, this deployment scheme also leads to severe resource underutilization. To address these limitations, we present MemEFS, an elastic in-memory runtime distributed file system. MemEFS is able to scale elastically, based on application storage demands, by acquiring or releasing resources when needed. Our evaluation shows that, while generating modest runtime overheads, MemEFS is able to increase the resource utilization efficiency by up to 65%.

An Elasticity Study of Distributed Graph Processing

“…For public infrastructure, elasticity could reduce operational costs by reducing resource waste [4], [5], and improve the ability to meet Quality-of-Service guarantees (such as high performance and availability) by using appropriate autoscaling policies [6]. For private infrastructure, elasticity could reduce operational costs by increasing resource utilization [7], or throttle throughput to meet demands across applications [8].…”

Elasticity in Graph Analytics? A Benchmarking Framework for Elastic Graph Processing

“…For achieving this, variants of elastic MapReduce [50], [51] have been developed. Furthermore, more domain-specific elasticity schemes have been proposed: machine learning on Spark [52], elastic stream processing [53] or even scientific workflows [22]. Similarly to such systems, JoyGraph supports user-defined policies for dynamically scaling the number of compute nodes during runtime.…”

“…For public infrastructure, elasticity could reduce operational costs by reducing resource waste [19], [20], and improve the ability to meet Quality-of-Service guarantees (such as high performance and availability) by using appropriate auto-scaling policies [21]. For private infrastructure, elasticity could reduce operational costs by increasing resource utilization [22], or throttle throughput to meet demands across applications [23].…”

Nswap2L

“…As the size of RAM continues to increase and its cost decreases over time, some recent work has referred to "RAM [as] the new disk," and disk as the new tape storage [23]. This trend in RAM capacity and cost has motivated filesystems based entirely in memory, including MemEFS [28] and RAMCloud [14]. Another study [10] concludes that data accessed every 5 hours should be placed in RAM, suggesting that many intermediate and temporary files created by large data-intensive computations could be stored in memory for their entire lifetime, avoiding secondary storage completely.…”