Transferring a petabyte in a day

Kettimuthu, Rajkumar; Liu, Zhengchun; Wheeler, David A.; Foster, Ian; Heitmann, Katrin; Cappello, Franck

doi:10.1016/j.future.2018.05.051

Cited by 33 publications

(12 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Because storage-storage and storage-WAN transfers tend to be nonblocking (e.g., Globus transfers are "fire and forget," burst buffer staging happens asynchronously, and HPSS data movement is managed through a batch queue at NERSC), we hypothesize that users may have less incentive to follow best practices for high-performance parallel I/O and instead initiate small-file transfers that are known to cause suboptimal endto-end performance [5], [23].…”

Section: Site-wide Transfer Behaviormentioning

confidence: 99%

“…High-performance computing (HPC) has historically been dominated by modeling and simulation workflows whose I/O needs are largely driven by checkpoint/restart. However, the role of HPC is rapidly expanding to include large-scale data analysis as a result of both increased data generation rates from modern scientific instruments and the emergence of artificial intelligence as a technique to rapidly extract insight from large volumes of data [1]- [5]. Hence, characterizing the requirements and performance of I/O in modern HPC centers now requires a more holistic examination of data movement.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Understanding Data Motion in the Modern HPC Data Center

Lockwood¹,

Snyder²,

Byna³

et al. 2019

2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW)

View full text Add to dashboard Cite

The utilization and performance of storage, compute, and network resources within HPC data centers have been studied extensively, but much less work has gone toward characterizing how these resources are used in conjunction to solve larger scientific challenges. To address this gap, we present our work in characterizing workloads and workflows at a data-center-wide level by examining all data transfers that occurred between storage, compute, and the external network at the National Energy Research Scientific Computing Center over a three-month period in 2019. Using a simple abstract representation of data transfers, we analyze over 100 million transfer logs from Darshan, HPSS user interfaces, and Globus to quantify the load on data paths between compute, storage, and the wide-area network based on transfer direction, user, transfer tool, source, destination, and time. We show that parallel I/O from user jobs, while undeniably important, is only one of several major I/O workloads that occurs throughout the execution of scientific workflows. We also show that this approach can be used to connect anomalous data traffic to specific users and file access patterns, and we construct time-resolved user transfer traces to demonstrate that one can systematically identify coupled data motion for individual workflows.

show abstract

Section: Site-wide Transfer Behaviormentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Understanding Data Motion in the Modern HPC Data Center

Lockwood¹,

Snyder²,

Byna³

et al. 2019

2019 IEEE/ACM Fourth International Parallel Data Systems Workshop (PDSW)

View full text Add to dashboard Cite

show abstract

“…Exascale systems are expected to generate an unprecedented amount of data in the upcoming decade. For example, cosmology applications such as HACC (Habib et al, 2013) generates 40 PB of data during a single trillion-particle simulation (Kettimuthu et al, 2018). The Community Earth System Model (CESM) which simulates the Earth’s past, present, and future climate states, produces 2.5 PB of raw data, which further leads to an estimate of about 12 PB of raw data output in CMIP6 (Paul et al, 2015).…”

Section: Introductionmentioning

confidence: 99%

Demystifying asynchronous I/O Interference in HPC applications

Tseng

Nicolae

Cappello

et al. 2021

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

With increasing complexity of HPC workflows, data management services need to perform expensive I/O operations asynchronously in the background, aiming to overlap the I/O with the application runtime. However, this may cause interference due to competition for resources: CPU, memory/network bandwidth. The advent of multi-core architectures has exacerbated this problem, as many I/O operations are issued concurrently, thereby competing not only with the application but also among themselves. Furthermore, the interference patterns can dynamically change as a response to variations in application behavior and I/O subsystems (e.g. multiple users sharing a parallel file system). Without a thorough understanding, I/O operations may perform suboptimally, potentially even worse than in the blocking case. To fill this gap, this paper investigates the causes and consequences of interference due to asynchronous I/O on HPC systems. Specifically, we focus on multi-core CPUs and memory bandwidth, isolating the interference due to each resource. Then, we perform an in-depth study to explain the interplay and contention in a variety of resource sharing scenarios such as varying priority and number of background I/O threads and different I/O strategies: sendfile, read/write, mmap/write underlining trade-offs. The insights from this study are important both to enable guided optimizations of existing background I/O, as well as to open new opportunities to design advanced asynchronous I/O strategies.

show abstract

“…In particular, data access and sharing has become more important for many HPC applications in the big data era. For example, a single cosmology application generates 20 PB of data, which is exchanged among dispersed computing facilities [1]. Identifying I/O and network bottlenecks should thus be one of the first-class requirements to improve efficiency and scalability in HPC systems.…”

Section: Introductionmentioning

confidence: 99%

“…Rather than simply relying on machine learning, we manually analyzed the log data to find out potential behavioral correlations that could cause performance bottlenecks. We present our initial observations from the analysis of one-week (January 1-6, 2018) HPC log data sets collected from a NERSC Cori HPC system 1 . The data set in this study includes the metadata server CPU load, file system logs sampled once every 5 seconds, and TCP connection records.…”

Section: Introductionmentioning

confidence: 99%

Spatio-Temporal Analysis of HPC I/O and Connection Data

Choi

Sim

2018

2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

The HPC system consists of a set of layers of software and hardware for I/O and networking. System logs are helpful resources to understand what is going on in the system. A challenge is that it is non-trivial to analyze the logs maintained in various levels of the stack. Independent analysis might lead to an incomplete conclusion due to the limited coverage of each log. This work takes a comprehensive approach to analysis that incorporates the logs in the multiple layers and components, in order to facilitate the detection of anomalous activities. This research aims to identify and predict potential performance bottlenecks in the HPC system, by capturing the temporal variation patterns from heterogeneous, high-dimensional, and non-linear log data. In this paper, we share our preliminary efforts for spatio-temporal analysis of HPC I/O and connection data, with our initial observations from the analysis of one-week HPC log data sets collected from one of NERSC systems.

show abstract

Transferring a petabyte in a day

Cited by 33 publications

References 16 publications

Understanding Data Motion in the Modern HPC Data Center

Understanding Data Motion in the Modern HPC Data Center

Demystifying asynchronous I/O Interference in HPC applications

Spatio-Temporal Analysis of HPC I/O and Connection Data

Contact Info

Product

Resources

About