Comprehensive Measurement and Analysis of the User-Perceived I/O Performance in a Production Leadership-Class Storage System

Wan, Lipeng; Wolf, Matthew; Wang, Feiyi; Choi, Jong Youl; Ostrouchov, George; Klasky, Scott

doi:10.1109/icdcs.2017.257

Cited by 19 publications

(7 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(3) Applying the above equation iteratively, we obtain the situation with any specific target bit-rate B * in Equation (2).…”

Section: ) Modeling Huffman Codingmentioning

confidence: 99%

“…Despite the ever-increasing computation power can be utilized to run the simulations nowadays, managing such large amounts of data remains challenging. It is impractical to save all the generated raw data to disk due to: (1) limited storage capacity even for large-scale parallel computers, and (2) the I/O bandwidth required to save this data to disk can create bottlenecks in the transmission [2]- [4].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Improving Prediction-Based Lossy Compression Dramatically via Ratio-Quality Modeling

Jin¹,

Di²,

Tian³

et al. 2021

Preprint

View full text Add to dashboard Cite

Error-bounded lossy compression is one of the most effective techniques for reducing scientific data sizes. However, the traditional trial-and-error approach used to configure lossy compressors for finding the optimal trade-off between reconstructed data quality and compression ratio is prohibitively expensive. To resolve this issue, we develop a general-purpose analytical ratio-quality model based on the prediction-based lossy compression framework, which can effectively foresee the reduced data quality and compression ratio, as well as the impact of lossy compressed data on post-hoc analysis quality. Our analytical model significantly improves the prediction-based lossy compression in three use-cases: (1) optimization of predictor by selecting the best-fit predictor; (2) memory compression with a target ratio; and (3) in-situ compression optimization by fine-grained tuning error-bounds for various data partitions. We evaluate our analytical model on 10 scientific datasets, demonstrating its high accuracy (93.47% accuracy on average) and low computational cost (up to 18.7× lower than the trialand-error approach) for estimating the compression ratio and the impact of lossy compression on post-hoc analysis quality. We also verify the high efficiency of our ratio-quality model using different applications across the three use-cases. In addition, our experiment demonstrates that our modeling based approach reduces the time to store the 3D RTM data with HDF5 by up to 3.4× with 128 CPU cores over the traditional solution.

show abstract

“…(3) Applying the above equation iteratively, we obtain the situation with any specific target bit-rate B * in Equation (2).…”

Section: ) Modeling Huffman Codingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Improving Prediction-Based Lossy Compression Dramatically via Ratio-Quality Modeling

Jin¹,

Di²,

Tian³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Such a large amount of data is often generated in a parallel manner from a scaling number of ranks, on which each holds a proportion of the data and must introduce an extra collective communication to dump the entire snapshot to the file system. This process takes an unprecedented challenge to I/O bandwidths and storage systems on today's HPC systems [7,30,54,55]. Therefore, it is urgent to develop effective data reduction methods to reduce the size of data movement between memories and storage systems such as parallel file systems.…”

Section: Introductionmentioning

confidence: 99%

Ceaz

Zhang

Jin

Geng

et al. 2022

Proceedings of the 36th ACM International Conference on Supercomputing

View full text Add to dashboard Cite

As HPC systems continue to grow to exascale, the amount of data that needs to be saved or transmitted is exploding. To this end, many previous works have studied using error-bounded lossy compressors to reduce the data size and improve the I/O performance. However, little work has been done for effectively offloading lossy compression onto FPGA-based SmartNICs to reduce the compression overhead. In this paper, we propose a hardware-algorithm codesign for an efficient and adaptive lossy compressor for scientific data on FPGAs (called CEAZ), which is the first lossy compressor that can achieve high compression ratios and throughputs simultaneously. Specifically, we propose an efficient Huffman coding approach that can adaptively update Huffman codewords online based on codewords generated offline, from a variety of representative scientific datasets. Moreover, we derive a theoretical analysis to support a precise control of compression ratio under an errorbounded compression mode, enabling accurate offline Huffman codewords generation. This also helps us create a fixed-ratio compression mode for consistent throughput. In addition, we develop an efficient compression pipeline by adopting cuSZ's dual-quantization algorithm to our hardware use cases. Finally, we evaluate CEAZ on five real-world datasets with both a single FPGA board and 128 nodes (to accelerate parallel I/O). Experiments show that CEAZ outperforms the second-best FPGA-based lossy compressor by 2.3× of throughput and 3.0× of ratio. It also improves MPI_File_write and MPI_Gather throughputs by up to 28.9× and 37.8×, respectively. CCS CONCEPTS• Computer systems organization → Reconfigurable computing.

show abstract

“…With the increase in scale of such simulations, saving all the raw data generated to disk becomes impractical due to: 1) limited storage capacity, and 2) the I/O bandwidth required to save this data to disk can create bottlenecks in the simulation [4,38,39] . For example, one Nyx simulation with a resolution of 4096 × 4096 × 4096 cells can generate up to 2.8 TB of data for a single snapshot; a total of 2.8 PB of disk storage is needed assuming running the simulation 5 times with 200 snapshots dumped per simulation.…”

Section: Introductionmentioning

confidence: 99%

Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling

Jin

Pulido

Grosset

et al. 2021

Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

Extreme-scale cosmological simulations have been widely used by today's researchers and scientists on leadership supercomputers. A new generation of error-bounded lossy compressors has been used in workflows to reduce storage requirements and minimize the impact of throughput limitations while saving large snapshots of high-fidelity data for post-hoc analysis. In this paper, we propose to adaptively provide compression configurations to compute partitions of cosmological simulations with newly designed postanalysis aware rate-quality modeling. The contribution is fourfold:(1) We propose a novel adaptive approach to select feasible error bounds for different partitions, showing the possibility and efficiency of adaptively configuring lossy compression for each partition individually. (2) We build models to estimate the overall loss of post-analysis result due to lossy compression and to estimate compression ratio, based on the property of each partition.(3) We develop an efficient optimization guideline to determine the best-fit configuration of error bounds combination in order to maximize the compression ratio under acceptable post-analysis quality loss. (4) Our approach introduces negligible overheads for feature extraction and error-bound optimization for each partition, enabling post-analysis-aware in situ lossy compression for cosmological simulations. Experiments show that our proposed models are highly accurate and reliable. Our fine-grained adaptive configuration approach improves the compression ratio of up to 73% on the tested datasets with the same post-analysis distortion with only 1% performance overhead.

show abstract

Comprehensive Measurement and Analysis of the User-Perceived I/O Performance in a Production Leadership-Class Storage System

Cited by 19 publications

References 14 publications

Improving Prediction-Based Lossy Compression Dramatically via Ratio-Quality Modeling

Improving Prediction-Based Lossy Compression Dramatically via Ratio-Quality Modeling

Ceaz

Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling

Contact Info

Product

Resources

About