Compression of <i>n</i>th‐order data arrays by B‐splines. Part 2: Application to second‐order FT‐IR spectra

Variable selection and compression are often used to produce more parsimonious regression models. But when they are applied directly to the original spectrum domain, it is not easy to determine the type of feature the selected variables represent. By performing variable selection in the wavelet domain we show that it is possible to identify important variables as being part of short-or large-scale features. Therefore, the suggested method is to extract information about the selected variables that otherwise would have been inaccessible. We are also able to obtain information about the location of these features in the original domain. In this article we demonstrate three types of variable selection methods applied to the wavelet domain: selection of optimal combination of scales, thresholding based on mutual information and truncation of weight vectors in the partial least squares (PLS) regression algorithm. We found that truncation of weight vectors in PLS was the most effective method for selecting variables. For the two experimental data sets tested we obtained approximately the same prediction error using less than 1% (for Data set 1) and 10% (for Data set 2) of the original variables. We also discovered that the selected variables were restricted to a limited number of wavelet scales. This information can be used to suggest whether the underlying features may be dominated by narrow (selective) peaks (indicated by variables in short wavelet scale regions) or by broader regions (indicated by variables in long wavelet scale regions). Thus, wavelet regression is here used as an extension of the more traditional Fourier regression (where the modelling is performed in the frequency domain without taking into consideration any of the information in the time domain). #

show abstract

“…B-splines [65,66]. The resulting coef®cients from a successful compression are less correlated than the original variables.…”

Section: Discussionmentioning

confidence: 99%

Variable selection in wavelet regression models

Alsberg

Woodward

Winson

et al. 1998

Analytica Chimica Acta

Self Cite

View full text Add to dashboard Cite

show abstract

“…Spectroscopists and analytical chemists have employed compression in the spectral domain to accelerate and improve data analysis. [1][2][3][4][5][6] The technique of variable selection is important for a 1 Sandia is a multiprogram laboratory operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy's National Nuclear Security Administration under contract DE-AC04-94AL85000. *mhvanbe@sandia.gov; phone 1-505-844-5443; www.sandia.gov †dpwoodb@sandia.gov variety of reasons.…”

Section: Spectral Compressionmentioning

confidence: 99%

An evaluation of algorithms and methods for compressing and decompressing atmospheric transmission data for use in at-sensor measurements

Benthem

Woodbury

2015

SPIE Proceedings

View full text Add to dashboard Cite

In this paper, we describe the use of various methods of one-dimensional spectral compression by variable selection as well as principal component analysis (PCA) for compressing multi-dimensional sets of spectral data. We have examined methods of variable selection such as wavelength spacing, spectral derivatives, and spectral integration error. After variable selection, reduced transmission spectra must be decompressed for use. Here we examine various methods of interpolation, e.g., linear, cubic spline and piecewise cubic Hermite interpolating polynomial (PCHIP) to recover the spectra prior to estimating at-sensor radiance. Finally, we compressed multi-dimensional sets of spectral transmittance data from moderate resolution atmospheric transmission (MODTRAN) data using PCA. PCA seeks to find a set of basis spectra (vectors) that model the variance of a data matrix in a linear additive sense. Although MODTRAN data are intricate and are used in nonlinear modeling, their base spectra can be reasonably modeled using PCA yielding excellent results in terms of spectral reconstruction and estimation of at-sensor radiance. The major finding of this work is that PCA can be implemented to compress MODTRAN data with great effect, reducing file size, access time and computational burden while producing high-quality transmission spectra for a given set of input conditions.

show abstract

“…The contribution of background, analytical signal and noise are generally assumed to occupy different frequency ranges by most authors [37][38][39][40] .With the WT transformation , the total spectra were split into several components with different frequency, and the t-value of high-scale approximation component and low-scale detail components were calculated, respectively. As to the fruit spectra data, the t-value curves corresponding to different scales of approximation are shown in Fig.…”

Section: Variable Contribution Of Multiscale Componentsmentioning

confidence: 99%

Effect of wavelet transform techniques upon the estimation of sugar content in apple with near-infrared spectroscopy

Liu

2004

SPIE Proceedings

View full text Add to dashboard Cite

Wavelet transform (WT) has proven a powerful and efficient tool for dealing with chemical data due to its characteristic of dual localization and has been widely used in analytical chemistry. This paper aims at serving three purposes: First, it gives a review of the applications of the wavelet transform in infrared spectroscopy; Second, it gives a quick summary of aspects and properties of wavelets and wavelet transforms which are needed in order to understand how to (pre-) process data from spectrometry with wavelet methods; Third, it shows on a typical example (apple NIR spectra) how wavelet transforms can be used in order to extract quantitative information. The sugar content of intact apple was measured by NIRS and analyzed by wavelet transform, which is a new development in signal treatment method in recent years. The results show that the spectra treated with wavelet transform indicate more effectively the relationship with sugar content in intact apple. Compared with original spectra, wavelet transform of three-size has the most marked relation with sugar content. The predicting precision of five-element regression is the best and the scale 3 is the best for its 0.904 correlation efficient of determination and the 0.777 in standard error of prediction which is less than that of primitive spectra. Therefore, the conclusion of improved predicting precision for quantitative detection of sugar content in intact apple with wavelet transform can be drawn.

show abstract

Compression of nth‐order data arrays by B‐splines. Part 2: Application to second‐order FT‐IR spectra

Cited by 29 publications

References 13 publications

Variable selection in wavelet regression models

Variable selection in wavelet regression models

An evaluation of algorithms and methods for compressing and decompressing atmospheric transmission data for use in at-sensor measurements

Effect of wavelet transform techniques upon the estimation of sugar content in apple with near-infrared spectroscopy

Contact Info

Product

Resources

About