ISABELA for effective in situ compression of scientific data

Abstract

Exploding dataset sizes from extreme-scale scientific simulations necessitates efficient data management and reduction schemes to mitigate I/O costs. With the discrepancy between I/O bandwidth and computational power, scientists are forced to capture data infrequently, thereby making data collection an inherently lossy process. Although data compression can be an effective solution, the random nature of real-valued scientific datasets renders lossless compression routines ineffective. These techniques also impose significant overhead during decompression, making them unsuitable for data analysis and visualization, which require repeated data access.To address this problem, we propose an effective method for In situ Sort-And-B-spline Error-bounded Lossy Abatement (ISABELA) of scientific data that is widely regarded as effectively incompressible. With ISABELA, we apply a pre-conditioner to seemingly random and noisy data along spatial resolution to achieve an accurate fitting model that guarantees a > 0.99 correlation with the original data. We further take advantage of temporal patterns in scientific data to compress data by 85%, while introducing only a negligible overhead on simulations in terms of runtime. ISABELA significantly outperforms existing lossy compression methods, such as wavelet compression, in terms of data reduction and accuracy.We extend upon our previous paper by additionally building a communication-free, scalable parallel storage framework on top of ISABELA-compressed data that is ideally suited for extreme-scale analytical processing. The basis for our storage framework is an inherently local decompression method (it need not decode the entire data), which allows for random access decompression and low-overhead task division that can be exploited over heterogeneous architectures. Furthermore, analytical operations such as correlation and query processing run quickly and accurately over data in the compressed space. 525In situ data processing-or processing the data in-tandem with the simulation by utilizing either the same compute nodes or the staging nodes-is emerging as a promising approach to address the I/O bottleneck [1]. To complement existing approaches, we introduce an effective method for In situ Sort-And-B-spline Error-bounded Lossy Abatement (ISABELA) of scientific data [2]. ISABELA is particularly designed for compressing spatio-temporal scientific data that is characterized as being inherently noisy and random-like and thus commonly believed to be incompressible [3]. In fact, the majority of the lossless compression techniques [4,5] are not only computationally intensive and therefore hardly suitable for in situ processing but also are unable to reduce such data by more than 10% of its original size (see Section 3).The intuition behind ISABELA stems from the following three observations. First, while being almost random and noisy in its natural form-when sorted-scientific data exhibits a very strong signal-to-noise ratio due to its monotonic and smooth behavior in its sorted form....

Z-checker: A framework for assessing lossy compression of scientific data

“…NUMARCK (Chen et al, 2014), for example, approximates the differences between snapshots by VQ. ISABELA (Lakshminarasimhan et al, 2013) converts the multidimensional data to a sorted data series and then performs B-spline interpolation. ZFP (Lindstrom, 2014) involves more complicated techniques such as fixed-point integer conversion, block transform, and binary representation analysis with bit-plane encoding.…”

“…They noted that the characteristics of the compression error must be carefully considered in the context of the underlying physics being modeled. Lakshminarasimhan et al (2013) proposed the ISABELA lossy compressor. It performs data compression by B-spline interpolation after sorting the data series.…”

Exascale Storage Systems - An Analytical Study of Expenses

“…In contrast to the file system level compression, however, the compression has to be performed by the applications themselves; even though the actual process is usually encapsulated in some I/O library such as HDF5, the involved compression overhead can negatively influence application performance. Application-specific knowledge can also be used by lossy compression schemes such as the floating point compressors proposed by Lindstrom and Isenburg [20], ISABELA [19] or GRIB and APAX to achieve larger gains for climate data [15]; this, however, forces the scientist to define boundaries for the tolerated loss in precision [15]. Recently, Baker et al presented an approach to determine the required precision for climate data; they achieve a compression ratio of 5 without noticeable impact on the scientific result [4].…”

Special Issue: Euro‐Par 2011

“…Their new contribution is to give preference to jobs with higher priority, as defined by a job selection function. They explore two different such functions and validate their benefits with simulation experiments.Topic 5 on Parallel and Distributed Data Management is represented by the paper ISABELA for effective in situ compression of scientific data, authored by Sriram Lakshminarasimhan, Neil Shah, Stephane Ethier, Seung-Hoe Ku, C. S. Chang, Scott Klasky, Robert Latham, Robert Ross, and Nagiza F. Samatova [2]. The authors address the fundamental problem of compressing the terabytes of numerical data produced by modern large-scale scientific simulations on high performance computing (HPC) systems.…”

“…Topic 16 on GPU and Accelerators Computing is represented by the paper Iterative sparse matrix-vector multiplication for accelerating the block Wiedemann algorithm over GF (2) on multi-GPU systems, authored by Bertil Schmidt, Hans Aribowo and Hoang Vu Dang [6]. Integer factorization constitutes the core of RSA cryptographic methods.…”