Assessing Differences in Large Spatio-temporal Climate Datasets with a New Python package

Pinard, Alexander; Hammerling, Dorit; Baker, Allison H.

doi:10.1109/bigdata50022.2020.9378100

Cited by 5 publications

(5 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Note that the default plot settings for the SSIM are the same as in Table I.) In particular, we plot the SSIM, DSSIM, and SF-DSSIM values that compare the original TS, PS, LHFLX, FLUT, and QFLX datasets to those datasets compressed with ten different levels of ZFP fixed-precision compression (p = 6, 8, 10,12,14,16,18,20,22,24). Recall that p = 6 is the most aggressive compressor option, and therefore should result in the smallest SSIM values for all variables, and p = 24 should result in values very close to one (or even equal to one).…”

Section: B Discussion Of Dssim Modificationsmentioning

confidence: 99%

“…This reduced computational cost is quite important when analysing large volumes of data. Note that the DSSIM is available in the LDCPY (Large Data Comparison for PYthon) package [24].…”

Section: Discussionmentioning

confidence: 99%

“…For these two variables, the two images being compared are very similar (SSIM = .99985 and SSIM = .99153 for TS and PRECT, respectively), and the differences cannot be seen for TS at this scale. The images and SSIM values are computed via the LDCPY 1 Python package [24], which uses the SSIM from SCIKIT-IMAGE and generates images using MATPLOTLIB [25] and CARTOPY [26]. Also note that the TS and PRECT data used in this section are included in the LDCPY package in data/cam-fv, from files zfp1e-1.TS.100days.nc, orig.TS.100days.nc, zfp1e-7.PRECT.60days.nc, and orig.PRECT.60days.nc.…”

Section: Ssim Value Dependenciesmentioning

confidence: 99%

See 2 more Smart Citations

DSSIM: a structural similarity index for floating-point data

Baker¹,

Pinard²,

Hammerling³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Data visualization is a critical component in terms of interacting with floating-point output data from large model simulation codes. Indeed, postprocessing analysis workflows on simulation data often generate a large number of images from the raw data, many of which are then compared to each other or to specified reference images. In this image-comparison scenario, image quality assessment (IQA) measures are quite useful, and the Structural Similarity Index (SSIM) continues to be a popular choice. However, generating large numbers of images can be costly, and plot-specific (but data independent) choices can affect the SSIM value. A natural question is whether we can apply the SSIM directly to the floating-point simulation data and obtain an indication of whether differences in the data are likely to impact a visual assessment, effectively bypassing the creation of a specific set of images from the data. To this end, we propose an alternative to the popular SSIM that can be applied directly to the floatingpoint data, which we refer to as the Data SSIM (DSSIM). While we demonstrate the usefulness of the DSSIM in the context of evaluating differences due to lossy compression on large volumes of simulation data from a popular climate model, the DSSIM may prove useful for many other applications involving simulation or image data.

show abstract

Section: B Discussion Of Dssim Modificationsmentioning

confidence: 99%

“…This reduced computational cost is quite important when analysing large volumes of data. Note that the DSSIM is available in the LDCPY (Large Data Comparison for PYthon) package [24].…”

Section: Discussionmentioning

confidence: 99%

Section: Ssim Value Dependenciesmentioning

confidence: 99%

See 1 more Smart Citation

DSSIM: a structural similarity index for floating-point data

Baker¹,

Pinard²,

Hammerling³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The expected error in the mean is zero with the bias-free rounding mode round-to-nearest and therefore the mean error is not explicitly analyzed here. Zfp compression can, however, introduce small errors in the mean 57,58 . The decimal error E dec is 59…”

Section: Preservation Of Gradientsmentioning

confidence: 99%

“…Owing to the rather logarithmic distribution of CAMS data (Supplementary Fig. 1), a log-preprocessing of the data is applied to prevent sign changes (including a flushing to zero) within the compression 57,58 . The error introduced by Zfp is approximately normally distributed and therefore usually yields higher maximum errors compared to round-to-nearest in float arithmetic, although median errors are comparable.…”

Section: Lossless Compressionmentioning

confidence: 99%

Compressing atmospheric data into its real information content

et al. 2021

View full text Add to dashboard Cite

Hundreds of petabytes are produced annually at weather and climate forecast centers worldwide. Compression is essential to reduce storage and to facilitate data sharing. Current techniques do not distinguish the real from the false information in data, leaving the level of meaningful precision unassessed. Here we define the bitwise real information content from information theory for the Copernicus Atmospheric Monitoring Service (CAMS). Most variables contain fewer than 7 bits of real information per value and are highly compressible due to spatio-temporal correlation. Rounding bits without real information to zero facilitates lossless compression algorithms and encodes the uncertainty within the data itself. All CAMS data are 17× compressed relative to 64-bit floats, while preserving 99% of real information. Combined with four-dimensional compression, factors beyond 60× are achieved. A data compression Turing test is proposed to optimize compressibility while minimizing information loss for the end use of weather and climate forecast data.

show abstract

Understanding the Effects of Modern Compressors on the Community Earth Science Model

Underwood

Bessac

et al. 2022

2022 IEEE/ACM 8th International Workshop on Data Analysis and Reduction for Big Scientific Data (DRBSD)

View full text Add to dashboard Cite

Assessing Differences in Large Spatio-temporal Climate Datasets with a New Python package

Cited by 5 publications

References 17 publications

DSSIM: a structural similarity index for floating-point data

DSSIM: a structural similarity index for floating-point data

Compressing atmospheric data into its real information content

Understanding the Effects of Modern Compressors on the Community Earth Science Model

Contact Info

Product

Resources

About