In search of an optimum sampling algorithm for prediction of soil properties from infrared spectra

Ng, Wartini; Minasny, Budiman; Malone, Brendan; Filippi, Patrick

doi:10.7717/peerj.5722

Cited by 37 publications

(23 citation statements)

References 36 publications

(49 reference statements)

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Better generalisation can be made by training the model in a larger data set. However, several studies demonstrated that the performance of the machine learning model did not increase significantly, or it even plateaued, as the calibration sample size increased (Figueroa et al, 2012;Ramirez-Lopez et al, 2014;Ng et al, 2018).…”

Section: Introductionmentioning

confidence: 99%

The influence of training sample size on the accuracy of deep learning models for the prediction of soil properties with near-infrared spectroscopy data

Minasny

Mendes³

et al. 2020

SOIL

Self Cite

View full text Add to dashboard Cite

Abstract. The number of samples used in the calibration data set affects the quality of the generated predictive models using visible, near and shortwave infrared (VIS–NIR–SWIR) spectroscopy for soil attributes. Recently, the convolutional neural network (CNN) has been regarded as a highly accurate model for predicting soil properties on a large database. However, it has not yet been ascertained how large the sample size should be for CNN model to be effective. This paper investigates the effect of the training sample size on the accuracy of deep learning and machine learning models. It aims at providing an estimate of how many calibration samples are needed to improve the model performance of soil properties predictions with CNN as compared to conventional machine learning models. In addition, this paper also looks at a way to interpret the CNN models, which are commonly labelled as a black box. It is hypothesised that the performance of machine learning models will increase with an increasing number of training samples, but it will plateau when it reaches a certain number, while the performance of CNN will keep improving. The performances of two machine learning models (partial least squares regression – PLSR; Cubist) are compared against the CNN model. A VIS–NIR–SWIR spectra library from Brazil, containing 4251 unique sites with averages of two to three samples per depth (a total of 12 044 samples), was divided into calibration (3188 sites) and validation (1063 sites) sets. A subset of the calibration data set was then created to represent a smaller calibration data set ranging from 125, 300, 500, 1000, 1500, 2000, 2500 and 2700 unique sites, which is equivalent to a sample size of approximately 350, 840, 1400, 2800, 4200, 5600, 7000 and 7650. All three models (PLSR, Cubist and CNN) were generated for each sample size of the unique sites for the prediction of five different soil properties, i.e. cation exchange capacity, organic carbon, sand, silt and clay content. These calibration subset sampling processes and modelling were repeated 10 times to provide a better representation of the model performances. Learning curves showed that the accuracy increased with an increasing number of training samples. At a lower number of samples (< 1000), PLSR and Cubist performed better than CNN. The performance of CNN outweighed the PLSR and Cubist model at a sample size of 1500 and 1800, respectively. It can be recommended that deep learning is most efficient for spectra modelling for sample sizes above 2000. The accuracy of the PLSR and Cubist model seems to reach a plateau above sample sizes of 4200 and 5000, respectively, while the accuracy of CNN has not plateaued. A sensitivity analysis of the CNN model demonstrated its ability to determine important wavelengths region that affected the predictions of various soil attributes.

show abstract

Section: Introductionmentioning

confidence: 99%

The influence of training sample size on the accuracy of deep learning models for the prediction of soil properties with near-infrared spectroscopy data

Minasny

Mendes³

et al. 2020

SOIL

Self Cite

View full text Add to dashboard Cite

show abstract

“…Visible and near infrared spectroscopy has been shown to successfully estimate various soil properties in many studies (e.g. Ng, Minasny, Malone, & Filippi, ; Singh et al, ; Stenberg, Rossel, Mouazen, & Wetterlind, ). The fundamental of soil reflectance spectroscopy is the stretching or bending of molecular bond that causes vibration to various degrees after absorbing light.…”

Section: Introductionmentioning

confidence: 99%

Evaluating a low‐cost portable NIR spectrometer for the prediction of soil organic and total carbon using different calibration models

Sharififar

Singh

Jones

et al. 2019

Soil Use and Management

Self Cite

View full text Add to dashboard Cite

This study aims to assess the performance of a low‐cost, micro‐electromechanical system‐based, near infrared spectrometer for soil organic carbon (OC) and total carbon (TC) estimation. TC was measured on 151 soil profiles up to the depth of 1 m in NSW, Australia, and from which a subset of 24 soil profiles were measured for OC. Two commercial spectrometers including the AgriSpecTM (ASD) and NeoSpectraTM (Neospectra) with spectral wavelength ranges of 350–2,500 and 1,300–2,500 nm, respectively, were used to scan the soil samples, according to the standard contact probe protocol. Savitzky–Golay smoothing filter and standard normal variate (SNV) transformation were performed on the spectral data for noise reduction and baseline correction. Three calibration models, including Cubist tree model, partial least squares regression (PLSR) and support vector machine (SVM), were assessed for the prediction of soil OC and TC using spectral data. A 10‐fold cross‐validation analysis was performed for evaluation of the models and devices accuracies. Results showed that Cubist model predicts OC and TC more accurately than PLSR and SVM. For OC prediction, Cubist showed R2 = 0.89 (RMSE = 0.12%) and R2 = 0.78 (RMSE = 0.16%) using ASD and NeoSpectra, respectively. For TC prediction, Cubist produced R2 = 0.75 (RMSE = 0.45%) and R2 = 0.70 (RMSE = 0.50%) using ASD and NeoSpectra, respectively. ASD performed better than NeoSpectra. However, the low‐cost NeoSpectra predictions were comparable to the ASD. These finding can be helpful for more efficient future spectroscopic prediction of soil OC and TC with less costly devices.

show abstract

“…Reducing the soil variability in the calibration dataset to only reflect the variability that will be encountered in the pilot study area would most likely improve predictions (Viscarra-Rossel et al, 2008). Also, we used a large sample size in our calibrations (on average n = 668; Supplemental Table S1), which may have created more reliable and representative models compared with models based on smaller datasets for similar-size geographical areas (Ng et al, 2018).…”

Section: Comparison With Other Studiesmentioning

confidence: 99%

Application of Mid‐Infrared Spectroscopy in Soil Survey

Seybold

Ferguson

Wysocki

et al. 2019

Soil Science Soc of Amer J

View full text Add to dashboard Cite

Development of more efficient (rapid) and cost-effective methodologies are needed in soil survey to meet the demand for quantitative data in digital soil mapping and updates. The objective of this study was to pilot the application of mid-infrared (MIR)-diffuse reflectance spectroscopy (DRS) coupled with partial least squares regression (PLSR) in a soil survey field office where soil samples are processed, MIR spectra are acquired, and predictions are obtained using calibration models developed and validated from the Kellogg Soil Survey Laboratory spectral library. Mid-infrared models were built for total C, organic C, CaCO 3 equivalent, total clay, cation exchange capacity, 1500 kPa water, and pH in water and CaCl 2 for Mollisols of the central United States. Validation results (from the MIR library) using Lin's concordance correlation (r c) of measured versus predicted values showed that most properties predicted very well (r c = 0.967-0.996), whereas models for total clay in B horizons and 1500 kPa water in B horizons predicted fairly well (r c = 0.844-0.955). Models for pH predicted the least well (r c = 0.750-0.921). The MIR-DRS coupled with PLSR was successful in predicting soil properties for completely independent samples that were collected, processed, and MIR scanned in a soil survey field office. Predicted results using r c ranged from 0.697 to 0.992, with pH in water having the lowest r c and CaCO 3 having the highest r c. All properties except pH had an acceptable level of accuracy for use in soil survey and a marginal level of acceptable accuracy for total clay. Direct calibration transfer was feasible.

show abstract

In search of an optimum sampling algorithm for prediction of soil properties from infrared spectra

Cited by 37 publications

References 36 publications

The influence of training sample size on the accuracy of deep learning models for the prediction of soil properties with near-infrared spectroscopy data

The influence of training sample size on the accuracy of deep learning models for the prediction of soil properties with near-infrared spectroscopy data

Evaluating a low‐cost portable NIR spectrometer for the prediction of soil organic and total carbon using different calibration models

Application of Mid‐Infrared Spectroscopy in Soil Survey

Contact Info

Product

Resources

About