A Year in the Life of a Parallel File System

Abstract

I/O performance is a critical aspect of data-intensive scientific computing. We seek to advance the state of the practice in understanding and diagnosing I/O performance issues through investigation of a comprehensive I/O performance data set that captures a full year of production storage activity at two leadership-scale computing facilities. We demonstrate techniques to identify regions of interest, perform focused investigations of both long-term trends and transient anomalies, and uncover the contributing factors that lead to performance fluctuation.We find that a year in the life of a parallel file system is comprised of distinct regions of long-term performance variation in addition to short-term performance transients. We demonstrate how systematic identification of these performance regions, combined with comprehensive analysis, allows us to isolate the factors contributing to different performance maladies at different time scales. From this, we present specific lessons learned and important considerations for HPC storage practitioners.

I/O Characterization and Performance Evaluation of BeeGFS for Deep Learning

“…In the tests, we increase the number of clients from 4 to 32 for both the models. We run the tests with Darshan-3.1.7, which is a well-established tool for HPC I/O tracing and characterization [20,21,37,38]. From the Darshan logs, we collect the file access patterns posed by the ImageNet data reader module in This phenomenon makes the application read less amount of data while the metadata overhead for file reading remains the same.…”

“…Besides, there has been another case study by Yu et al [50] on the performance of Lustre over Quadrics and InfiniBand using sequential and parallel I/O, metadata and application benchmarks. While quite a number of recent studies [33,36,40,[46][47][48][49]52] have demonstrated different techniques to evaluate the I/O performance of PFSs in supercomputing facilities, many other studies [34,39,43] have emphasized on PFS, application and data API tuning and optimization to acquire better I/O throughput on HPC systems. Although many research attempts have been taken for I/O performance analysis of different PFSs, there is a lack of understanding on the characteristics of BeeGFS I/O and metadata performance, particularly its capability of handling the workloads posed by DL applications.…”

“…Its frequent development to adapt to cutting-edge technologies makes it more promising [1, 3,5,9]. Even though there have been lots of research contributions on the evaluation of different PFSs [33,36,40,[46][47][48][50][51][52], not many studies have been performed on BeeGFS. Hence, there is a need for a systematic I/O characterization and performance analysis of BeeGFS for both system practitioners and application users who would like to explore its use on leadership computers.…”

Toward Understanding I/O Behavior in HPC Workflows

“…This framework includes access to privileged monitoring information from I/O subsystems and vendor APIs, but it also makes use of application data collected by using Darshan. By continuously monitoring a data center over time, one can, for example, detect performance regressions [24]. TOKIO specifically.…”

Understanding Data Motion in the Modern HPC Data Center

“…A large body of knowledge exists on data motion between compute and storage from the perspectives of both applications and systems [6]- [11] because this form of I/O blocks forward progress on running jobs. However, data-driven workflows rely on data movement not only between compute and storage systems, but also between storage systems and between storage and external networks as depicted in Figure 1.…”