ARCHIE: Data Analysis Acceleration with Array Caching in Hierarchical Storage

Dong, Bo; Wang, Teng; Tang, Houjun; Koziol, Quincey; Wu, Kesheng; Byna, Suren

doi:10.1109/bigdata.2018.8622616

Cited by 10 publications

(3 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In read caching, we have implemented caching and prefetching chunks of data based on the history of HDF5 chunk accesses [5] . The cached chunks are stored as binary files in a faster persistent storage device, such as an SSD-based burst buffer.…”

Section: Read Caching In Data Elevatormentioning

confidence: 99%

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Byna

Breitenfeld

Dong

et al. 2020

J. Comput. Sci. Technol.

Self Cite

View full text Add to dashboard Cite

Scientific applications at exascale generate and analyze massive amounts of data. A critical requirement of these applications is the capability to access and manage this data efficiently on exascale systems. Parallel I/O, the key technology enables moving data between compute nodes and storage, faces monumental challenges from new application, memory, and storage architectures considered in the designs of exascale systems. As the storage hierarchy is expanding to include node-local persistent memory, burst buffers, etc., as well as disk-based storage, data movement among these layers must be efficient. Parallel I/O libraries of the future should be capable of handling file sizes of many terabytes and beyond. In this paper, we describe new capabilities we have developed in Hierarchical Data Format version 5 (HDF5), the most popular parallel I/O library for scientific applications. HDF5 is one of the most used libraries at the leadership computing facilities for performance parallel I/O on existing HPC systems. The state-of-the-art features we describe include: Virtual Object Layer (VOL), Data Elevator, asynchronous I/O, full-featured single-writer and multiple-reader (Full SWMR), and parallel querying. In this paper, we introduce these features, their implementations, and the performance and feature benefits to applications and other libraries.

show abstract

Section: Read Caching In Data Elevatormentioning

confidence: 99%

ExaHDF5: Delivering Efficient Parallel I/O on Exascale Computing Systems

Byna

Breitenfeld

Dong

et al. 2020

J. Comput. Sci. Technol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recent works such as Hermes [19], UniviStor [20] and ARCHIE [18] provide buffering solution on all stages of storage layers. Hermes offers three data placement policies to manage caching and prefetching of flat data files.…”

Section: Related Workmentioning

confidence: 99%

“…On read, they need to populate the Burst Buffer through manual stage-in operations. Since prefetching data to an intermediate storage is also necessary for many postprocessing applications, systems that support one-way read operation, such as ARCHIE, have also been proposed [18]. Hermes and UniviStor are new systems that offer multi-tiered buffering solutions [19], [20].…”

Section: Introductionmentioning

confidence: 99%

MLBS: Transparent Data Caching in Hierarchical Storage for Out-of-Core HPC Applications

Alturkestani

Tonellot²,

Ltaief

et al. 2019

2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC)

View full text Add to dashboard Cite

Out-of-core simulation systems produce and/or consume a massive amount of data that cannot fit on a single compute node memory and that usually needs to be read and/or written back and forth during computation. I/O data movement may thus represent a bottleneck in large-scale simulations. To increase I/O bandwidth, high-end supercomputers are equipped with hierarchical storage subsystems such as node-local and remote-shared NVMe and SSD-based Burst Buffers. Advanced caching systems have recently been developed to efficiently utilize the multi-layered nature of the new storage hierarchy. Utilization of software components results in more efficient data accesses, at the cost of reduced computation kernel performance and limited numbers of simultaneous applications that can utilize the additional storage layers. We introduce MultiLayered Buffer Storage (MLBS), a data object container that provides novel methods for caching and prefetching data in out-of-core scientific applications to perform asynchronously expensive I/O operations on systems equipped with hierarchical storage. The main idea consists in decoupling I/O operations from computational phases using dedicated hardware resources to perform expensive context switches. MLBS monitors I/O traffic in each storage layer allowing fair utilization of shared resources while controlling the impact on kernels' performance. By continually prefetching up and down across all hardware layers of the memory/storage subsystems, MLBS transforms the original I/O-bound behavior of evaluated applications and shifts it closer to a memorybound regime. Our evaluation on a Cray XC40 system for a representative I/O-bound application, seismic inversion, shows that MLBS outperforms state-of-the-art filesystems, i.e., Lustre, Data Elevator and DataWarp by 6.06X, 2.23X, and 1.90X, respectively.

show abstract