Predicting file lifetimes for data placement in multi-tiered storage systems for HPC

Abstract

The emergence of Exascale machines in HPC will have the foreseen consequence of putting more pressure on the storage systems in place, not only in terms of capacity but also bandwidth and latency. With limited budget we cannot imagine using only storage class memory, which leads to the use of a heterogeneous tiered storage hierarchy. In order to make the most efficient use of the high performance tier in this storage hierarchy, we need to be able to place user data on the right tier and at the right time. In this paper, we assume a 2-tier storage hierarchy with a high performance tier and a high capacity archival tier. Files are placed on the high performance tier at creation time and moved to capacity tier once their lifetime expires (that is once they are no more accessed). The main contribution of this paper lies in the design of a file lifetime prediction model solely based on its path based on the use of Convolutional Neural Network. Results show that our solution strikes a good trade-off between accuracy and under-estimation. Compared to previous work, our model made it possible to reach an accuracy close to previous work (around 98.60% compared to 98.84%) while reducing the underestimations by almost 10x to reach 2.21% (compared to 21.86%). The reduction in underestimations is crucial as it avoids misplacing files in the capacity tier while they are still in use.

Efficient Prefetching and Client-Side Caching Algorithms for Improving the Performance of Read Operations in Distributed File Systems

“…Several state-of-the-art multi-level caching techniques were proposed in the literature [24]- [27] which mainly concentrates on caching the data based on prediction by analyzing the characteristics of I/O accesses. The authors of [28] predicted the life time of files by analyzing the access frequency of the files. The authors in [29] proposed WorkflowRL method which manages the data in the multilevel storage systems based on reinforcement learning.…”

Studying Scientific Data Lifecycle in On-demand Distributed Storage Caches

“…Knowledge regarding file lifetimes can enable the development and implementation of cache eviction policies superior to the Least Recently Used (LRU) policy. [12,15].…”

Studying Scientific Data Lifecycle in On-demand Distributed Storage Caches

“…This work could also be expanded upon by developing machine learning models that can predict when a certain file is likely to be evicted from the cache. Such models have been shown to be effective in creating more precise and detailed file lifetime data [15]. A more effective model could inform design choices for better distributed storage caches.…”

“…Thus, it is beneficial to know how long files tend to remain open. Knowledge regarding file lifetimes can enable the development and implementation of cache eviction policies superior to the Least Recently Used (LRU) policy [12,15]…”