Evaluating image quality measures to assess the impact of lossy data compression applied to climate simulation data

Baker, Allison H.; Hammerling, Dorit; Turton, Terece L.

doi:10.1111/cgf.13707

Cited by 26 publications

(33 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reason 1D SSIM always approaches to 1 is that the mean, standard deviation and covariance are always very similar between the original data and decompressed data. Some recent studies [35], [38] on visualization shows that 2D SSIM could be more accurate than PSNR in some cases, so we suggest to use both PSNR and 2D SSIM in the evaluation of lossy compression quality. Table V also contains compression ratios of five state-ofthe-art lossless compressors.…”

Section: Evaluation Of Lossy Compressorsmentioning

confidence: 99%

SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors

Zhao,

Di,

Liang

et al. 2021

Preprint

View full text Add to dashboard Cite

Efficient error-controlled lossy compressors are becoming critical to the success of today's large-scale scientific applications because of the ever-increasing volume of data produced by the applications. In the past decade, many lossless and lossy compressors have been developed with distinct design principles for different scientific datasets in largely diverse scientific domains. In order to support researchers and users assessing and comparing compressors in a fair and convenient way, we establish a standard compression assessment benchmark -Scientific Data Reduction Benchmark (SDRBench) 1 . SDRBench contains a vast variety of real-world scientific datasets across different domains, summarizes several critical compression quality evaluation metrics, and integrates many state-of-the-art lossy and lossless compressors. We demonstrate evaluation results using SDRBench and summarize six valuable takeaways that are helpful to the in-depth understanding of lossy compressors.

show abstract

Section: Evaluation Of Lossy Compressorsmentioning

confidence: 99%

SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors

Zhao,

Di,

Liang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…These scores cover important variables in various large scale regions, such as 2m-temperature over Europe or horizontal wind speed at different vertical levels in the Southern Hemisphere. With a similar motivation as in Baker et al 52 , we suggest assessing the efficiency of climate data compression using similar scores, which have to be passed similar to a Turing test 33,53 . The compressed forecast data should be indistinguishable from the uncompressed data in all of these score tests, or at least indistinguishable from the current compression method while allowing higher compression factors.…”

Section: A Data Compression Turing Testmentioning

confidence: 99%

Compressing atmospheric data into its real information content

Kloewer

Razinger

Dominguez

et al. 2021

Preprint

View full text Add to dashboard Cite

Hundreds of petabytes of data are produced annually at weather and climate forecast centres worldwide. Compression is inevitable to reduce storage and to facilitate data sharing. Current techniques do not distinguish the real from the false information in data. We define the bitwise real information content from information theory for data from the Copernicus Atmospheric Monitoring Service (CAMS). Most variables contain less than 7 bits of real information per value, which are also highly compressible due to spatio-temporal correlation. Rounding bits without real information to zero facilitates lossless compression algorithms and encodes the uncertainty within the data itself. The entire CAMS data is compressed by a factor of 17x, relative to 64-bit floats, while preserving 99% of real information. Combined with 4-dimensional compression to exploit the spatio-temporal correlation, factors beyond 60x are achieved without an increase in forecast errors. A data compression Turing test is proposed to optimize compressibility while minimizing information loss for the end use of weather and climate forecast data.

show abstract

“…Lossy compression does not always outperform lossless compression in case of constrained error margins [93]. Lossy compression is common in computer graphics [7,16] and used by many visualizations primarily meant to be interpreted by humans but are not fed back to numerical computations.…”

Section: Lossy Compressionmentioning

confidence: 99%

State of the Art and Future Trends in Data Reduction for High-Performance Computing

2020

JSFI

View full text Add to dashboard Cite

Research into data reduction techniques has gained popularity in recent years as storage capacity and performance become a growing concern. This survey paper provides an overview of leveraging points found in high-performance computing (HPC) systems and suitable mechanisms to reduce data volumes. We present the underlying theories and their application throughout the HPC stack and also discuss related hardware acceleration and reduction approaches. After introducing relevant use-cases, an overview of modern lossless and lossy compression algorithms and their respective usage at the application and file system layer is given. In anticipation of their increasing relevance for adaptive and in situ approaches, dimensionality reduction techniques are summarized with a focus on non-linear feature extraction. Adaptive approaches and in situ compression algorithms and frameworks follow. The key stages and new opportunities to deduplication are covered next. An unconventional but promising method is recomputation, which is proposed at last. We conclude the survey with an outlook on future developments.

show abstract

Evaluating image quality measures to assess the impact of lossy data compression applied to climate simulation data

Cited by 26 publications

References 42 publications

SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors

SDRBench: Scientific Data Reduction Benchmark for Lossy Compressors

Compressing atmospheric data into its real information content

State of the Art and Future Trends in Data Reduction for High-Performance Computing

Contact Info

Product

Resources

About