NUMARCK: Machine Learning Algorithm for Resiliency and Checkpointing

Chen, Zhengzhang; Son, Seung Woo; Hendrix, William; Agrawal, Ankit; Liao, Wei-keng; Choudhary, Alok

doi:10.1109/sc.2014.65

Cited by 54 publications

(30 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…State-of-the-art lossy compressors often combine multiple strategies, such as vector quantization (VQ), orthogonal transform, prediction, and analysis of floating-point binary representation (BA). NUMARCK (Chen et al, 2014), for example, approximates the differences between snapshots by VQ. ISABELA (Lakshminarasimhan et al, 2013) converts the multidimensional data to a sorted data series and then performs B-spline interpolation.…”

Section: Research Background and Design Motivationmentioning

confidence: 99%

Z-checker: A framework for assessing lossy compression of scientific data

Tao

Guo

et al. 2017

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

Because of the vast volume of data being produced by today's scientific simulations and experiments, lossy data compressor allowing user-controlled loss of accuracy during the compression is a relevant solution for significantly reducing the data size. However, lossy compressor developers and users are missing a tool to explore the features of scientific data sets and understand the data alteration after compression in a systematic and reliable way. To address this gap, we have designed and implemented a generic framework called Z-checker. On the one hand, Z-checker combines a battery of data analysis components for data compression. On the other hand, Z-checker is implemented as an open-source community tool to which users and developers can contribute and add new analysis components based on their additional analysis demands. In this article, we present a survey of existing lossy compressors. Then, we describe the design framework of Z-checker, in which we integrated evaluation metrics proposed in prior work as well as other analysis tools. Specifically, for lossy compressor developers, Z-checker can be used to characterize critical properties (such as entropy, distribution, power spectrum, principal component analysis, and autocorrelation) of any data set to improve compression strategies. For lossy compression users, Z-checker can detect the compression quality (compression ratio and bit rate) and provide various global distortion analysis comparing the original data with the decompressed data (peak signal-to-noise ratio, normalized mean squared error, rate-distortion, rate-compression error, spectral, distribution, and derivatives) and statistical analysis of the compression error (maximum, minimum, and average error; autocorrelation; and distribution of errors). Z-checker can perform the analysis with either coarse granularity (throughout the whole data set) or fine granularity (by user-defined blocks), such that the users and developers can select the best fit, adaptive compressors for different parts of the data set. Z-checker features a visualization interface displaying all analysis results in addition to some basic views of the data sets such as time series. To the best of our knowledge, Z-checker is the first tool designed to assess lossy compression comprehensively for scientific data sets.

show abstract

Section: Research Background and Design Motivationmentioning

confidence: 99%

Z-checker: A framework for assessing lossy compression of scientific data

Tao

Guo

et al. 2017

The International Journal of High Performance Computing Applica

View full text Add to dashboard Cite

show abstract

“…It is also worth noting that this workflow is essentially a materials science adaptation of existing similar workflows of data-driven analytics in other domains, as most of the advanced techniques for big data management and informatics come from the field of computer science and more specifically high-performance data mining, [42][43][44][45][46][47][48][49][50] via applications in many different [64][65][66][67][68] and social media analytics, 69-71 among many others.…”

Section: Knowledge Discovery Workflow For Materials Informaticsmentioning

confidence: 99%

Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science

2016

Self Cite

View full text Add to dashboard Cite

Our ability to collect “big data” has greatly surpassed our capability to analyze it, underscoring the emergence of the fourth paradigm of science, which is data-driven discovery. The need for data informatics is also emphasized by the Materials Genome Initiative (MGI), further boosting the emerging field of materials informatics. In this article, we look at how data-driven techniques are playing a big role in deciphering processing-structure-property-performance relationships in materials, with illustrative examples of both forward models (property prediction) and inverse models (materials discovery). Such analytics can significantly reduce time-to-insight and accelerate cost-effective materials discovery, which is the goal of MGI.

show abstract

“…The PSNR estimation will be −10 · log 10 ( n i=1 δ 2 i ) + 20 · log 10 V R + 10 · log 10 (6n). The bin size can be estimated by the clustering-based approximation approach (proposed by Chen et al [21]), such as the K-means cluster algorithm, whose time overhead is expensive.…”

Section: Detailed Analysis Of Three Vector Quantization Casesmentioning

confidence: 99%

Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP

Tao

Liang

et al. 2019

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

With ever-increasing volumes of scientific data produced by high-performance computing applications, significantly reducing data size is critical because of limited capacity of storage space and potential bottlenecks on I/O or networks in writing/reading or transferring data. SZ and ZFP are two leading BSD licensed open source C/C++ libraries for compressed floating-point arrays that support high throughput read and write random access. However, their performance is not consistent across different data sets and across different fields of some data sets, which raises the need for an automatic online (during compression) selection between SZ and ZFP, with minimal overhead. In this paper, the automatic selection optimizes the rate-distortion, an important statistical quality metric based on the signal-to-noise ratio. To optimize for rate-distortion, we investigate the principles of SZ and ZFP. We then propose an efficient online, low-overhead selection algorithm that predicts the compression quality accurately for two compressors in early processing stages and selects the best-fit compressor for each data field. We implement the selection algorithm into an open-source library, and we evaluate the effectiveness of our proposed solution against plain SZ and ZFP in a parallel environment with 1,024 cores. Evaluation results on three data sets representing about 100 fields show that our selection algorithm improves the compression ratio up to 70% with the same level of data distortion because of very accurate selection (around 99%) of the bestfit compressor, with little overhead (less than 7% in the experiments).

show abstract

NUMARCK: Machine Learning Algorithm for Resiliency and Checkpointing

Cited by 54 publications

References 18 publications

Z-checker: A framework for assessing lossy compression of scientific data

Z-checker: A framework for assessing lossy compression of scientific data

Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science

Optimizing Lossy Compression Rate-Distortion from Automatic Online Selection between SZ and ZFP

Contact Info

Product

Resources

About