Hyperspectral Estimation of Soil Organic Matter Content using Different Spectral Preprocessing Techniques and PLSR Method

Abstract

Soil organic matter (SOM) is the main source of soil nutrients, which are essential for the growth and development of agricultural crops. Hyperspectral remote sensing is one of the most efficient ways of estimating the SOM content. Visible, near infrared, and mid-infrared reflectance spectroscopy, combined with the partial least squares regression (PLSR) method is considered to be an effective way of determining soil properties. In this study, we used 54 different spectral pretreatments to preprocess soil spectral data. These spectral pretreatments were composed of three denoising methods, six data transformations, and three dimensionality reduction methods. The three denoising methods included no denoising (ND), Savitzky–Golay denoising (SGD), and wavelet packet denoising (WPD). The six data transformations included original spectral data, R; reciprocal, 1/R; logarithmic, log(R); reciprocal logarithmic, log(1/R); first derivative, R’; and first derivative of reciprocal, (1/R)’. The three dimensionality reduction methods included no dimensionality reduction (NDR), sensitive waveband dimensionality reduction (SWDR), and principal component analysis (PCA) dimensionality reduction (PCADR). The processed spectra were then employed to construct PLSR models for predicting the SOM content. The main results were as follows—(1) the wavelet packet denoising (WPD)-R’ and WPD-(1/R)’ data showed stronger correlations with the SOM content. Furthermore, these methods could effectively limit the correlation between the adjacent bands and, thus, prevent “overfitting”. (2) Of the 54 pretreatments investigated, WPD-(1/R)’-PCADR yielded the model with the highest accuracy and stability. (3) For the same denoising method and spectral transformation data, the accuracy of the SOM content estimation model based on SWDR was higher than that of the model based on NDR. Furthermore, the accuracy in the case of PCADR was higher than that for SWDR. (4) Dimensionality reduction was effective in preventing data overfitting. (5) The quality of the spectral data could be improved and the accuracy of the SOM content estimation model could be enhanced effectively, by using some appropriate preprocessing methods (one combining WPD and PCADR in this study).

Exploring Appropriate Preprocessing Techniques for Hyperspectral Soil Organic Matter Content Estimation in Black Soil Area

“…First derivative (FD) and second derivative (SD) are commonly used among data transformation methods in the preprocessing of spectral data. This is related to the fact that FD and SD are able to extract hidden information in the spectrum and eliminate baseline effect [24][25][26]. The positive effect of integral derivatives on SOM content estimation has been confirmed [25,27].…”

“…In Figure 2, the correlation curves between SOM content and reflectance shows that SGF can effectively improve the correlation, while the correlation curve after WPT basically coincides with the original correlation curve. The advantages of the SGF were not apparent in the results obtained by Shen et al [25]. Determining appropriate smoothing window size was essential for removing noise and retain information of spectral data [37].…”

“…This is related to the fact that FD and SD are able to extract hidden information in the spectrum and eliminate baseline effect [24][25][26]. The positive effect of integral derivatives on SOM content estimation has been confirmed [25,27]. Nevertheless, integer derivatives lack the sensitivity of gradual changes in slit and curvature, so that noise introduction and information loss will be caused [28][29][30].…”

“…This is related to the noise introduced by SD while separating the absorption peak. Moreover, the spectrum after FD and SD forms obvious sharp peaks [25]. Among 12 combinations, 10 combinations achieved the optimal performance at order 0.2-0.8.…”

“…Dimensionality reduction of spectral data can reduce data redundancy and improve the accuracy of SOM content estimation models [25]. Principal component analysis (PCA) is the most widely used dimensionality reduction method for spectral data.…”