Improving the prediction performance of a large tropical vis‐NIR spectroscopic soil library from Brazil by clustering into smaller subsets or use of data mining calibration techniques

Abstract

Effective agricultural planning requires basic soil information. In recent decades visible near-infrared diffuse reflectance spectroscopy (vis-NIR) has been shown to be a viable alternative for rapidly analysing soil properties. We studied 7172 samples of seven different soil types collected from several regions of Brazil and varying in organic matter (OM) (0.2-10.3%) and clay content (0.2-99.0%). The aim was to explore the possibility of enhancing the performance of vis-NIR data in predicting organic matter and clay content in this library by dividing it into smaller sub-libraries on the basis of their vis-NIR spectra. We used partial least square regression (PLSR) models on the sub-libraries and compared the results with PLSR and two non-linear calibration techniques, boosted regression trees (BT) and support vector machines (SVM) applied to the whole library. The whole library calibrations for clay performed well (ME (modelling efficiency) > 0.82; RMSE (root mean squared error) < 10.9%), reflecting the influence of the direct spectral responses of this property in the vis-NIR range. Calibrations for OM were reasonably good, especially in view of the very small variation in this property (ME > 0.60; RMSE < 0.55%). The best results were, however, found when dividing the large library into smaller subsets by using variation in the mean-normalized or first derivative spectra. This divided the global data set into clusters that were more uniform in mineralogy, regardless of geographical origin, and improved predictive performance. The best clustering method improved the RMSE in the validation to 8.6% clay and 0.47% OM, which corresponds to a 21% and 15% reduction, respectively, as compared with whole library PLSR. For the whole library, SVM performed almost equally well, reducing RMSE to 8.9% clay and 0.48% OM.

Agricultural Soil Spectral Response and Properties Assessment: Effects of Measurement Protocol and Data Mining Technique

“…It is clear that VIS-NIR-SWIR spectra contain useful information that can be used to derive estimates of soil properties. For example, absorption features in the VIS-NIR wavelength (400-1000 nm) are characteristics of the presence of soil carbon and iron oxide [34,35,[48][49][50], and those in the SWIR (1000-2500 nm) are from water, clay minerals and organic matter [16,51]. The important spectra absorption features through the use of some data mining algorithms have been studied by Viscarra Rossel and Behrens [15] and Gholizadeh et al [17].…”

“…For example, Gholizadeh et al [14] indicated that the 1st derivative preprocessing method gave the best prediction of heavy metals in the Czech Republic mining areas, in comparison to 2nd derivative, multiplicative scatter correction (MSC), standard normal variate (SNV) and continuum removal (CR). Viscarra Rossel and Behrens [15] and Araujo et al [16] applied partial least square regression (PLSR), boosted regression trees (BRT) and support vector machine regression (SVMR) methods for the prediction of clay; SVMR offered the most successful prediction model due to its ability to solve the multivariate calibration problems and to reduce problems with heterogeneity and non-linearity. However, in a study by Gholizadeh et al [17], the memory based learning (MBL) technique outperformed PLSR, BRT and SVMR in soil texture prediction, which can be attributed to the selection of more appropriate neighbours to calibrate local models, as well as the inclusion of more suitable neighbours in each local model as a source of additional predictor variables [18].…”

A Memory-Based Learning Approach as Compared to Other Data Mining Algorithms for the Prediction of Soil Texture Using Diffuse Reflectance Spectra

“…2016, 8,341 3 of 17 been given to the prediction of other soil textural classes, including silt and sand. The use of VNIR/SWIR reflectance spectroscopy offers a lower precision than for clay, especially with particular chemometric algorithms.…”

“…The method makes multiple predictions that are based on resampling and weighting and belongs to the group of ensemble techniques [66]. It has the ability to take in a large number of weak relationships in a predictive model, and it is not sensitive to outliers in the calibration dataset [8]. Following Friedman [66], boosted models can be stated in the general form:…”

“…It can be said that the group of positive peaks near 900 nm may represent absorption caused by electronic transitions in goethite due to Laporte forbidden transitions [74]. The small absorption bands occurring near 1200 nm, 1400 nm and 1900 nm may be due to the vibrational combinations and overtones of molecular water contained in various locations in minerals [8,79]. To be more accurate, the group of peaks near 1400 nm may be attributed to the first overtone of the O-H stretch; the peaks near 1700 nm, due to the first overtone of the C-H stretch; the prominent group of peaks near 1900 nm may be related to H-O-H bend with the O-H stretches [77].…”