DAOS for Extreme-scale Systems in Scientific Applications

Breitenfeld, M. Scot; Fortner, Neil; Henderson, Jordan; Soumagne, Jerome; Chaarawi, Mohamad; Lombardi, Johann; Koziol, Quincey

doi:10.48550/arxiv.1712.00423

Cited by 1 publication

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While emerging storage technologies [39]- [41] that aim to elevate I/O bottlenecks, and projects such as SAGE [42] and DAOS [43], are being intensively studied, I/O will likely remain a challenging aspect when deploying ML workload. By enabling fine-grained profiling and tracing capability, we also enable the opportunity for automated decision making and auto-tuning in the future.…”

Section: Discussionmentioning

confidence: 99%

tf-Darshan: Understanding Fine-grained I/O Performance in Machine Learning Workloads

Chien

Podobas

2020

2020 IEEE International Conference on Cluster Computing (CLUSTER)

View full text Add to dashboard Cite

Machine Learning applications on HPC systems have been gaining popularity in recent years. The upcoming large scale systems will offer tremendous parallelism for training through GPUs. However, another heavy aspect of Machine Learning is I/O, and this can potentially be a performance bottleneck. TensorFlow, one of the most popular Deep-Learning platforms, now offers a new profiler interface and allows instrumentation of TensorFlow operations. However, the current profiler only enables analysis at the TensorFlow platform level and does not provide system-level information. In this paper, we extend TensorFlow Profiler and introduce tf-Darshan, both a profiler and tracer, that performs instrumentation through Darshan. We use the same Darshan shared instrumentation library and implement a runtime attachment without using a system preload. We can extract Darshan profiling data structures during TensorFlow execution to enable analysis through the TensorFlow profiler. We visualize the performance results through TensorBoard, the web-based TensorFlow visualization tool. At the same time, we do not alter Darshan's existing implementation. We illustrate tf-Darshan by performing two case studies on ImageNet image and Malware classification. We show that by guiding optimization using data from tf-Darshan, we increase POSIX I/O bandwidth by up to 19% by selecting data for staging on fast tier storage. We also show that Darshan has the potential of being used as a runtime library for profiling and providing information for future optimization.

show abstract