Comparing different data preprocessing methods for monitoring soil heavy metals based on soil spectral features

Abstract

Gholizadeh A., Borůvka L., Saberioon M.M., Kozák J., Vašát R., Němeček K. (2015): Comparing different data preprocessing methods for monitoring soil heavy metals based on soil spectral features. Soil & Water Res., 10: 218-227.The lands near mining industries in the Czech Republic are subjected to soil pollution with heavy metals. Excessive heavy metal concentrations in soils not only dramatically impact the soil quality, but also due to their persistent nature and indefinite biological half-lives, potentially toxic metals can accumulate in the food chain and can eventually endanger human health. Monitoring and spatial information of these elements require a large number of samples and cumbersome and time-consuming laboratory measurements. A faster method has been developed based on a multivariate calibration procedure using support vector machine regression (SVMR) with cross-validation, to establish a relationship between reflectance spectra in the visible-near infrared (Vis-NIR) region and concentration of Mn, Cu, Cd, Zn, and Pb in soil. Spectral preprocessing methods, first and second derivatives (FD and SD), standard normal variate (SNV), multiplicative scatter correction (MSC), and continuum removal (CR) were employed after smoothing with Savitzky-Golay to improve the robustness and performance of the calibration models. According to the criteria of maximal coefficient of determination (R 2 cv ) and minimal root mean square error of prediction in cross-validation (RMSEP cv ), the SVMR algorithm with FD preprocessing was determined as the best method for predicting Cu, Mn, Pb, and Zn concentration, whereas the SVMR model with CR preprocessing was chosen as the final method for predicting Cd. Overall, this study indicated that the Vis-NIR reflectance spectroscopy technique combined with a continuously enriched soil spectral library as well as a suitable preprocessing method could be a nondestructive alternative for monitoring of the soil environment. The future possibilities of multivariate calibration and preprocessing with real-time remote sensing data have to be explored.

A Memory-Based Learning Approach as Compared to Other Data Mining Algorithms for the Prediction of Soil Texture Using Diffuse Reflectance Spectra

“…Spectra preprocessing algorithms entailed a range of mathematical techniques for refining light scattering in spectral reflectance measurements and data improvement before the data were used in calibration models. The first derivative transformation, which was utilized in this study, is very efficient for eliminating baseline offset and, according to some researchers, gives the best results and uppermost accuracy among other algorithms [28,41,53]. In this study, before all further spectra treatments, the noisy parts of the spectra, ranges 350-399 nm and 2450-2500 nm, were removed, and the spectra were subjected to Savitzky-Golay smoothing with a second-order polynomial fit and 11 smoothing points [18,54] for eliminating the artificial noise caused by the spectroradiometer device.…”

“…The question arises: why another study on different calibration approaches? As shown by Gholizadeh et al [10,41], choosing the most robust calibration technique can help to achieve a more reliable and accurate prediction model. Moreover, different studies reveal different results, because the nature of the target function has a significant effect on the performance of the different prediction approaches.…”

“…In order to create a robust prediction model and to discover the impression of the spectral sampling interval on the prediction accuracy, Savitzky-Golay smoothing with second-order polynomial fit and 11 smoothing points with subsequent first derivative preprocessing technique were applied prior to model calibration [41,83,84]. Smoothed spectra only, by Savitzky-Golay filter, as well as smoothed and preprocessed spectra, using Savitzky-Golay plus the first derivative, of all The spectra have absorption peaks overlapping near 430 nm and 530 nm in the Visible (VIS) region, which state the presence of iron oxides, and are caused by paired and single Fe 3+ electron transitions to a higher energy state [74][75][76][77].…”

Agricultural Soil Spectral Response and Properties Assessment: Effects of Measurement Protocol and Data Mining Technique

“…Suitable data preprocessing, calibration and validation strategies, which frequently differ for users and operators, to calibrate soil prediction models, influence the final model too. For example, Gholizadeh et al [14] indicated that the 1st derivative preprocessing method gave the best prediction of heavy metals in the Czech Republic mining areas, in comparison to 2nd derivative, multiplicative scatter correction (MSC), standard normal variate (SNV) and continuum removal (CR). Viscarra Rossel and Behrens [15] and Araujo et al [16] applied partial least square regression (PLSR), boosted regression trees (BRT) and support vector machine regression (SVMR) methods for the prediction of clay; SVMR offered the most successful prediction model due to its ability to solve the multivariate calibration problems and to reduce problems with heterogeneity and non-linearity.…”

Assessment of soil heavy metal pollution in a former mining area - before and after the end of mining activities

“…The transport of heavy metals in soil is the result of processes between soil and metal components, which include processes of physical, chemical, and biological nature (Violante et al 2008). However, soil is not only a passive acceptor of heavy metals, polluted soils become a source of contamination for other environmental components and the food chain (Gholizadeh et al 2015). In addition, heavy metals are non-degradable and persistent, their presence in soil is stable and doi: 10.17221/107/2016-SWR long-term (Lizárraga-Mendiola et al 2009).…”