Mathematical Data Preprocessing

Duckworth, James

doi:10.2134/agronmonogr44.c6

Cited by 25 publications

(28 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The optimum number of terms in the PLS1 calibration models was indicated by the lowest number of factors that gave the minimum value of the prediction residual error sum of squares in crossvalidation in order to avoid over-fitting in the models (15). The ATR-MIR spectral data was processed using the secondderivative Savitzky-Golay (second derivative, 40 smoothing points and second polynomial order) in order to remove and correct for baseline effects (16). The second derivative is a measure of the change in the slope of the curve, ignoring the offset, and is very effective in removing both baseline offset and slope from a spectrum (16).…”

Section: Discussionmentioning

confidence: 99%

“…The ATR-MIR spectral data was processed using the secondderivative Savitzky-Golay (second derivative, 40 smoothing points and second polynomial order) in order to remove and correct for baseline effects (16). The second derivative is a measure of the change in the slope of the curve, ignoring the offset, and is very effective in removing both baseline offset and slope from a spectrum (16). Statistics calculated for the calibrations included the coefficient of determination in cross-validation R 2 CV À Á , the standard error of cross-validation (SECV), bias and slope.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Feasibility study on the use of attenuated total reflectance mid-infrared spectroscopy for the analysis of malt quality parameters in wort

et al. 2014

View full text Add to dashboard Cite

The feasibility of using attenuated total reflectance in combination with mid-infrared (MIR) spectroscopy for the analysis of malt quality parameters in wort, such as free α-amino acid (FAN), apparent attenuation limit (AAL), soluble protein (SP) and viscosity (VIS), was evaluated. The coefficients of determination in cross-validation R 2 CV À Á and the standard error of crossvalidation were 0.81 and 10.3 mg L À1 for FAN, 0.80 and 1.49% for AAL, 0.78 and 0.18% for SP and 0.69 and 0.13 cP for VIS. Good calibration and validation statistics were obtained for AAL. However, for nitrogenous compounds the MIR predictions can only be used as approximate estimates of the true concentrations, for example to rank samples (low, medium, high) in the context of a breeding programme.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Discussionmentioning

confidence: 99%

Feasibility study on the use of attenuated total reflectance mid-infrared spectroscopy for the analysis of malt quality parameters in wort

et al. 2014

View full text Add to dashboard Cite

show abstract

“…We used the sgolayfilt algorithm from the signal R package for the SG filtering (adjusted for second-order polynomial fit with 30 smoothing points). For more detail about the pretreatment, the packages used can be found in [31][32][33][34].…”

Section: Spectra Pretreatment and Prediction Model Developmentmentioning

confidence: 99%

Comparison of Field and Laboratory Wet Soil Spectra in the Vis-NIR Range for Soil Organic Carbon Prediction in the Absence of Laboratory Dry Measurements

et al. 2020

View full text Add to dashboard Cite

Spectroscopy has demonstrated the ability to predict specific soil properties. Consequently, it is a promising avenue to complement the traditional methods that are costly and time-consuming. In the visible-near infrared (Vis-NIR) region, spectroscopy has been widely used for the rapid determination of organic components, especially soil organic carbon (SOC) using laboratory dry (lab-dry) measurement. However, steps such as collecting, grinding, sieving and soil drying at ambient (room) temperature and humidity for several days, which is a vital process, make the lab-dry preparation a bit slow compared to the field or laboratory wet (lab-wet) measurement. The use of soil spectra measured directly in the field or on a wet sample remains challenging due to uncontrolled soil moisture variations and other environmental conditions. However, for direct and timely prediction and mapping of soil properties, especially SOC, the field or lab-wet measurement could be an option in place of the lab-dry measurement. This study focuses on comparison of field and naturally acquired laboratory measurement of wet samples in Visible (VIS), Near-Infrared (NIR) and Vis-NIR range using several pretreatment approaches including orthogonal signal correction (OSC). The comparison was concluded with the development of validation models for SOC prediction based on partial least squares regression (PLSR) and support vector machine (SVMR). Nonetheless, for the OSC implementation, we use principal component regression (PCR) together with PLSR as SVMR is not appropriate under OSC. For SOC prediction, the field measurement was better in the VIS range with R2CV = 0.47 and RMSEPcv = 0.24, while in Vis-NIR range the lab-wet measurement was better with R2CV = 0.44 and RMSEPcv = 0.25, both using the SVMR algorithm. However, the prediction accuracy improves with the introduction of OSC on both samples. The highest prediction was obtained with the lab-wet dataset (using PLSR) in the NIR and Vis-NIR range with R2CV = 0.54/0.55 and RMSEPcv = 0.24. This result indicates that the field and, in particular, lab-wet measurements, which are not commonly used, can also be useful for SOC prediction, just as the lab-dry method, with some adjustments.

show abstract

“…In the present study, the number of removed outliers was four. Then, the first derivative calculation was used as a spectra preprocessing technique, the transformation of which is very effective for removing baseline offset [11,39].…”

Section: Spectra Preprocessing For Selected Data-mining Techniquesmentioning

confidence: 99%

Examining the Performance of PARACUDA-II Data-Mining Engine versus Selected Techniques to Model Soil Carbon from Reflectance Spectra

et al. 2018

View full text Add to dashboard Cite

The monitoring and quantification of soil carbon provide a better understanding of soil and atmosphere dynamics. Visible-near-infrared-short-wave infrared (VIS-NIR-SWIR) reflectance spectroscopy can quantitatively estimate soil carbon content more rapidly and cost-effectively compared to traditional laboratory analysis. However, effective estimation of soil carbon using reflectance spectroscopy to a great extent depends on the selection of a suitable preprocessing sequence and data-mining algorithm. Many efforts have been dedicated to the comparison of conventional chemometric techniques and their optimization for soil properties prediction. Instead, the current study focuses on the potential of the new data-mining engine PARACUDA-II®, recently developed at Tel-Aviv University (TAU), by comparing its performance in predicting soil oxidizable carbon (Cox) against common data-mining algorithms including partial least squares regression (PLSR), random forests (RF), boosted regression trees (BRT), support vector machine regression (SVMR), and memory based learning (MBL). To this end, 103 soil samples from the Pokrok dumpsite in the Czech Republic were scanned with an ASD FieldSpec III Pro FR spectroradiometer in the laboratory under a strict protocol. Spectra preprocessing for conventional data-mining techniques was conducted using Savitzky-Golay smoothing and the first derivative method. PARACUDA-II®, on the other hand, operates based on the all possibilities approach (APA) concept, a conditional Latin hypercube sampling (cLHs) algorithm and parallel programming, to evaluate all of the potential combinations of eight different spectral preprocessing techniques against the original reflectance and chemical data prior to the model development. The comparison of results was made in terms of the coefficient of determination (R2) and root-mean-square error of prediction (RMSEp). Results showed that the PARACUDA-II® engine performed better than the other selected regular schemes with R2 value of 0.80 and RMSEp of 0.12; the PLSR was less predictive compared to other techniques with R2 = 0.63 and RMSEp = 0.29. This can be attributed to its capability to assess all the available options in an automatic way, which enables the hidden models to rise up and yield the best available model.

show abstract

Mathematical Data Preprocessing

Cited by 25 publications

References 6 publications

Feasibility study on the use of attenuated total reflectance mid-infrared spectroscopy for the analysis of malt quality parameters in wort

Feasibility study on the use of attenuated total reflectance mid-infrared spectroscopy for the analysis of malt quality parameters in wort

Comparison of Field and Laboratory Wet Soil Spectra in the Vis-NIR Range for Soil Organic Carbon Prediction in the Absence of Laboratory Dry Measurements

Examining the Performance of PARACUDA-II Data-Mining Engine versus Selected Techniques to Model Soil Carbon from Reflectance Spectra

Contact Info

Product

Resources

About