No abstract

Exploding dataset sizes from extreme-scale scientific simulations necessitates efficient data management and reduction schemes to mitigate I/O costs. With the discrepancy between I/O bandwidth and computational power, scientists are forced to capture data infrequently, thereby making data collection an inherently lossy process. Although data compression can be an effective solution, the random nature of real-valued scientific datasets renders lossless compression routines ineffective. These techniques also impose significant overhead during decompression, making them unsuitable for data analysis and visualization, which require repeated data access.To address this problem, we propose an effective method for In situ Sort-And-B-spline Error-bounded Lossy Abatement (ISABELA) of scientific data that is widely regarded as effectively incompressible. With ISABELA, we apply a pre-conditioner to seemingly random and noisy data along spatial resolution to achieve an accurate fitting model that guarantees a > 0.99 correlation with the original data. We further take advantage of temporal patterns in scientific data to compress data by 85%, while introducing only a negligible overhead on simulations in terms of runtime. ISABELA significantly outperforms existing lossy compression methods, such as wavelet compression, in terms of data reduction and accuracy.We extend upon our previous paper by additionally building a communication-free, scalable parallel storage framework on top of ISABELA-compressed data that is ideally suited for extreme-scale analytical processing. The basis for our storage framework is an inherently local decompression method (it need not decode the entire data), which allows for random access decompression and low-overhead task division that can be exploited over heterogeneous architectures. Furthermore, analytical operations such as correlation and query processing run quickly and accurately over data in the compressed space. 525In situ data processing-or processing the data in-tandem with the simulation by utilizing either the same compute nodes or the staging nodes-is emerging as a promising approach to address the I/O bottleneck [1]. To complement existing approaches, we introduce an effective method for In situ Sort-And-B-spline Error-bounded Lossy Abatement (ISABELA) of scientific data [2]. ISABELA is particularly designed for compressing spatio-temporal scientific data that is characterized as being inherently noisy and random-like and thus commonly believed to be incompressible [3]. In fact, the majority of the lossless compression techniques [4,5] are not only computationally intensive and therefore hardly suitable for in situ processing but also are unable to reduce such data by more than 10% of its original size (see Section 3).The intuition behind ISABELA stems from the following three observations. First, while being almost random and noisy in its natural form-when sorted-scientific data exhibits a very strong signal-to-noise ratio due to its monotonic and smooth behavior in its sorted form....

Current peta-scale data analytics frameworks suffer from a significant performance bottleneck due to an imbalance between their enormous computational power and limited I/O bandwidth. Using data compression schemes to reduce the amount of I/O activity is a promising approach to addressing this problem. In this paper, we propose a hybrid framework for interleaving I/O with data compression to achieve improved I/O throughput side-by-side with reduced dataset size. We evaluate several interleaving strategies, present theoretical models, and evaluate the efficiency and scalability of our approach through comparative analysis. With our theoretical model, considering 19 real-world scientific datasets both from the public domain and peta-scale simulations, we estimate that the hybrid method can result in a 12 to 46% increase in throughput on hard-to-compress scientific datasets. At the reported peak bandwidth of 60 GB/s of uncompressed data for a current, leadership-class parallel I/O system, this translates into an effective gain of 7 to 28 GB/s in aggregate throughput.