Revisiting I/O behavior in large-scale storage systems

Patel, Tirthak; Byna, Suren; Lockwood, Glenn K.; Tiwari, Devesh

doi:10.1145/3295500.3356183

Cited by 44 publications

(24 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As exemplified in the introduction, the effect of conflicting accesses on the I/O subsystem is a well-known problem in HPC infrastructures [11,23,31]. Recently the load imbalance In this context, the I/O performance is difficult to model due to the complex interaction between different interfering applications.…”

Section: Related Workmentioning

confidence: 99%

Mapping and scheduling HPC applications for optimizing I/O

Carretero

Jeannot

Pallez

et al. 2020

Proceedings of the 34th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

In HPC platforms, concurrent applications are sharing the same file system. This can lead to conflicts, especially as applications are more and more data intensive. I/O contention can represent a performance bottleneck. The access to bandwidth can be split in two complementary yet distinct problems. The mapping problem and the scheduling problem. The mapping problem consists in selecting the set of applications that are in competition for the I/O resource. The scheduling problem consists then, given I/O requests on the same resource, in determining the order to these accesses to minimize the I/O time. In this work we propose to couple a novel bandwidth-aware mapping algorithm to I/O list-scheduling policies to develop a cross-layer optimization solution.We study this solution experimentally using an I/O middleware: CLARISSE. We show that naive policies such as FIFO perform relatively well in order to schedule I/O movements, and that the important part to reduce congestion lies mostly on the mapping part. We evaluate the algorithm that we propose using a simulator that we validated experimentally. This evaluation shows important gains for the simple, bandwidth-aware mapping solution that we provide compared to its non bandwidth-aware counterpart. The gains are both in terms of machine efficiency (makespan) and application efficiency (stretch). This stresses even more the importance of designing efficient, bandwidth-aware mapping strategies to alleviate the cost of I/O congestion.

show abstract

Section: Related Workmentioning

confidence: 99%

Mapping and scheduling HPC applications for optimizing I/O

Carretero

Jeannot

Pallez

et al. 2020

Proceedings of the 34th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

show abstract

“…Many works [1], [15], [32] have characterized the performance of various file systems and their impact on general applications. Yu et al [32] characterized Lustre parallel file system atop a Cray XT supercomputer.…”

Section: Related Workmentioning

confidence: 99%

“…Yu et al [32] characterized Lustre parallel file system atop a Cray XT supercomputer. Patel et al [1] performed an in-depth statistical analysis of a year-long collection of storage data to study the patterns on accessing large parallel file systems on HPC systems. Patel et al [15] classifies files and then studied the file reuse and access characteristics in files on petabyte-scale systems on production supercomputers.…”

Section: Related Workmentioning

confidence: 99%

“…Machine Learning (ML) has gained considerable attention in recent years for its potential to drive advances in selfdriving cars, drug development, and image processing. Demands for ML workloads on many HPC systems are increasing overtime [1], [2]. New ML methods, such as Deep-Learning (DL) [3], involve a computational intensive minimization process.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloads

Chien

Podobas

2020

2020 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

Machine Learning applications on HPC systems have been gaining popularity in recent years. The upcoming large scale systems will offer tremendous parallelism for training through GPUs. However, another heavy aspect of Machine Learning is I/O, and this can potentially be a performance bottleneck. TensorFlow, one of the most popular Deep-Learning platforms, now offers a new profiler interface and allows instrumentation of TensorFlow operations. However, the current profiler only enables analysis at the TensorFlow platform level and does not provide system-level information. In this paper, we extend TensorFlow Profiler and introduce tf-Darshan, both a profiler and tracer, that performs instrumentation through Darshan. We use the same Darshan shared instrumentation library and implement a runtime attachment without using a system preload. We can extract Darshan profiling data structures during TensorFlow execution to enable analysis through the TensorFlow profiler. We visualize the performance results through TensorBoard, the web-based TensorFlow visualization tool. At the same time, we do not alter Darshan's existing implementation. We illustrate tf-Darshan by performing two case studies on ImageNet image and Malware classification. We show that by guiding optimization using data from tf-Darshan, we increase POSIX I/O bandwidth by up to 19% by selecting data for staging on fast tier storage. We also show that Darshan has the potential of being used as a runtime library for profiling and providing information for future optimization.

show abstract

“…In contrast, the insystem storage layers provide lower I/O access latencies and higher I/O throughput for emerging AI/ML workloads. These two systems are representative of the diversity of production HPC systems since [6], [13], [20] ✓ ✓ ✓ [10], [19], [22] ✓ ✓ ✓ [9], [28] ✓…”

Section: Introductionmentioning

confidence: 99%

Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production Load

Bez

Karimi

Paul

et al. 2022

Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing

Self Cite

View full text Add to dashboard Cite

Scientific computing workloads at HPC facilities have been shifting from traditional numerical simulations to AI/ML applications for training and inference while processing and producing everincreasing amounts of scientific data. To address the growing need for increased storage capacity, lower access latency, and higher bandwidth, emerging technologies such as non-volatile memory are integrated into supercomputer I/O subsystems. With these emerging trends, we need a better understanding of the multilayer supercomputer I/O systems and ways to use these subsystems efficiently. In this work, we study the I/O access patterns and performance characteristics of two representative supercomputer I/O subsystems. Through an extensive analysis of year-long I/O logs on each system, we report new observations in I/O reads and writes, unbalanced use of storage system layers, and new trends in user behaviors at the HPC I/O middleware stack.

show abstract

Revisiting I/O behavior in large-scale storage systems

Cited by 44 publications

References 34 publications

Mapping and scheduling HPC applications for optimizing I/O

Mapping and scheduling HPC applications for optimizing I/O

tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloads

Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production Load

Contact Info

Product

Resources

About