Lossless data compression studies for NOAA hyperspectral environmental suite using 3D integer wavelet transforms with 3D embedded zerotree coding

A method 7 combining the empirical mode decomposition (EMD) 6 and the principal component analysis (PCA) was recently proposed for lossless compression of ultraspectral sounder data. In that method, data residual is obtained via the linear regression of the data against m intrinsic mode functions (IMFs) which are obtained from the EMD of the data mean, followed by the linear regression of the IMF regression error against a truncated number, n, of the corresponding principal components (PCs). In this paper we show that this two-stage (m IMFs + n PCs) linear transform approach is not as good as its counterpart two-stage (m PCs + n PCs) linear transform approach in terms of data residual and compression ratio of ultraspectral data, given the same number of IMFs and PCs used respectively at the first stage, followed by the same number of PCs used at the second stage. Mathematically, the two-stage (m PCs + n PCs) linear transform approach is equivalent to a single linear transform with (m + n) PCs. In other words, the simple PCA compression method outperforms this combined EMD and PCA compression method. This is expected because the PCA (a.k.a. the Karhunen-Loève transform or the Hotelling transform) is known to be the optimal linear transform in the sense of minimizing the mean squared error. Our compression experiment confirms our conjecture that the role of EMD in the EMD + PCA approach is not only redundant but also degrades the compression ratios as compared to the PCA approach. We also compare the effect of using a partial set of 1502 channels against using the full set of 2107 channels, and show that the EMD + PCA approach does not break the 3:1 lossless compression barrier when compressing the full set of 2107 channels in the standard ultraspectral test data set. Our lossless PCA approach 19 is the first to break the 3:1 compression barrier for the full set of this class of data.

show abstract

“…AIRS 1 , CrIS 2 , IASI 3 , GIFTS 4 , HES 5 etc.) has made improved weather prediction and climate monitoring possible.…”

Section: Introductionmentioning

confidence: 99%

<title>On empirical mode decomposition for ultraspectral sounder data compression</title>

2005

Self Cite

View full text Add to dashboard Cite

show abstract

“…Wavelet zerotree coding schemes such as 2D EZW 7 and 2D SPIHT 8 , and 2D JPEG2000 9 belong to the general class of wavelet based data compression schemes, whereas 2D JPEG-LS 10 and 2D CALIC 11 belong to the class of predictor based compression schemes. We extend both 2D EZW and 2D SPIHT to 3D for any size of 3D satellite data, each of whose dimensions need not be divisible by 2 N , where N is the layers of the wavelet decomposition being performed 5,17 . For other 2D schemes such as JPEG2000, JPEG-LS, and CALIC, we transform the 3D data into 2D via a continuous scan.…”

Section: Introductionmentioning

confidence: 99%

“…Better inference of the atmospheric, cloud and surface parameters is made possible by contemporary and future hyperspectral sounders such as Atmospheric Infrared Sounder (AIRS) 1 , Crosstrack Infrared Sounder (CrIS) 2 , Infrared Atmospheric Sounding Interferometer (IASI) 3 , Geosynchronous Imaging Fourier Transform Spectrometer (GIFTS) 4 , and Hyperspectral Environmental Suite (HES) 5 . The price for this is a large volume of 3D data produced by hyperspectral infrared sounders.…”

Section: Introductionmentioning

confidence: 99%

Lossless data compression for infrared hyperspectral sounders: an update

Huang

Ahuja

et al. 2004

SPIE Proceedings

Self Cite

View full text Add to dashboard Cite

The compression of hyperspectral sounder data is beneficial for more efficient archive and transfer given its large 3-D volume. Moreover, since physical retrieval of geophysical parameters from hyperspectral sounder data is a mathematically ill-posed problem that is sensitive to the error of the data, lossless or near-lossless compression is desired. This paper provides an update into applications of state-of-the-art 2D and 3D lossless compression algorithms such as 3D EZW, 3D SPIHT, 2D JPEG2000, 2D JPEG-LS and 2D CALIC for hyperspectral sounder data. In addition, in order to better explore the correlations between the remote spectral regions affected by the same type of atmospheric absorbing constituents or clouds, the Bias-Adjusted Reordering (BAR) scheme is presented which reorders the data such that the bias-adjusted distance between any two neighboring vectors is minimized. This scheme coupled with any of the state-of-the-art compression algorithms produces significant compression gains.

show abstract