Robust variable selection in the framework of classification with label noise and outliers: Applications to spectroscopic data in agri-food

Cappozzo, Andrea; Duponchel, Ludovic; Greselin, Francesca; Murphy, Brendan

doi:10.1016/j.aca.2021.338245

Cited by 8 publications

(4 citation statements)

References 56 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In partial least squares regression (PLSR) models, the response is usually required to be continuous and obey the normal distribution [19] . Outliers of Y would produce leverage lead to deviation in variable extraction and distort the outcome and accuracy of a regression [20,21] . The Monte Carlo sampling (MCS) method could help to reduce the risk that the masking effect brings about and provide a feasible way to detect different kinds of outliers using the distribution of prediction errors of a test sample resulting from a population of sub-models [22][23][24] .…”

Section: Introductionmentioning

confidence: 99%

Modeling method for SSC prediction in pomelo using Vis-NIRS with wavelength selection and latent variable updating

Tian,

Wang,

et al. 2024

International Journal of Agricultural and Biological Engineering

View full text Add to dashboard Cite

The aim of this study was in-line, rapid, and non-destructive detection for soluble solid content (SSC) in pomelos using visible and near-infrared spectroscopy (Vis-NIRS). However, the large size and thick rind of pomelo affect the stability of spectral acquisition and the biological variabilities affect the robustness of models. Given these issues, in this study, an efficient prototype in-line detection system in transmittance mode was designed and evaluated in comparison with an off-line detection system. Data from the years 2019 and 2020 were used for modeling and the external validation data were obtained by the inline detection system in 2021. The wavelength selection methods of changeable size moving window (CSMW), random frog (RF), and competitive adaptive reweighted sampling (CARS) were used to improve the prediction accuracy of partial least squares regression (PLSR) models. The best performance of internal prediction was obtained by CARS-PLSR and the determination coefficient of prediction ( ), root mean square error of prediction (RMSEP), and residual predictive deviation (RPD) were 0.958, 0.204%, and 4.821, respectively. However, all models obtained large prediction biases in external validation. The latent variable updating (LVU) method was proposed to update models and improve the performance in external validation. Ten samples from the external validation set were randomly selected to update the models. Compared with the recalibration method, LVU could effectively modify the original models which matched the SSC range of the external validation set. The CSMW-PLSR models were more robust in external validations. The off-line model with LVU performed best with a root mean square error of validation (RMSEV) of 0.599% and the in-line model with recalibration obtained RMSEV of 0.864%. These results demonstrated the application potential of the transmittance Vis-NIRS for in-line rapid prediction of SSC in pomelos and the modeling and updating methods could be applied to samples with biological variabilities.

show abstract

Section: Introductionmentioning

confidence: 99%

Modeling method for SSC prediction in pomelo using Vis-NIRS with wavelength selection and latent variable updating

Tian,

Wang,

et al. 2024

International Journal of Agricultural and Biological Engineering

View full text Add to dashboard Cite

show abstract

“…Real‐world data are often subject to errors or noise in a variety of fields 1,2 . Among them, datasets coming from chemical analyses and procedures in diverse applications, such as agri‐food or medicine, may be particularly prone to imperfections 3,4 . These errors proceed from different sources associated with chemical practice, including human action and the instruments used, among others 5,6 .…”

Section: Introductionmentioning

confidence: 99%

Noise simulation in classification with the noisemodel R package: Applications analyzing the impact of errors with chemical data

Sáez

2023

Journal of Chemometrics

View full text Add to dashboard Cite

Classification datasets created from chemical processes can be affected by errors, which impair the accuracy of the models built. This fact highlights the importance of analyzing the robustness of classifiers against different types and levels of noise to know their behavior against potential errors. In this context, noise models have been proposed to study noise‐related phenomenology in a controlled environment, allowing errors to be introduced into the data in a supervised manner. This paper introduces the noisemodel R package, which contains the first extensive implementation of noise models for classification datasets, proposing it as support tool to analyze the impact of errors related to chemical data. It provides 72 noise models found in the specialized literature that allow errors to be introduced in different ways in classes and attributes. Each of them is properly documented and referenced, unifying their results through a specific S3 class, which benefits from customized print, summary and plot methods. The usage of the package is illustrated through four application examples considering real‐world chemical datasets, where errors are prone to occur. The software presented will help to deepen the understanding of the problem of noisy chemical data, as well as to develop new robust algorithms and noise preprocessing methods properly adapted to different types of errors in this scenario.

show abstract

“…Raman spectra acquisition generates noise and outliers, which the Huber loss function can address by combining squared and absolute errors [ 22 ]. To further improve the regression model’s robustness, we propose a new Gaussian kernel Huber loss function that more accurately measures the deep learning error’s nonlinearity and handles noise or outliers better [ 23 ]. Combining intelligent algorithms with neural network algorithms has become a main research direction in recent years to address the difficulty of parameter control in neural network building [ 24 ].…”

Section: Introductionmentioning

confidence: 99%

Raman spectroscopy-based prediction of ofloxacin concentration in solution using a novel loss function and an improved GA-CNN model

Ma,

Shi,

Huang

et al. 2023

BMC Bioinformatics

View full text Add to dashboard Cite

Background A Raman spectroscopy method can quickly and accurately measure the concentration of ofloxacin in solution. This method has the advantages of accuracy and rapidity over traditional detection methods. However, the manual analysis methods for the collected Raman spectral data often ignore the nonlinear characteristics of the data and cannot accurately predict the concentration of the target sample. Methods To address this drawback, this paper proposes a novel kernel-Huber loss function that combines the Huber loss function with the Gaussian kernel function. This function is used with an improved genetic algorithm-convolutional neural network (GA-CNN) to model and predict the Raman spectral data of different concentrations of ofloxacin in solution. In addition, the paper introduces recurrent neural networks (RNN), long short-term memory (LSTM), bidirectional long short-term memory (BiLSTM) and gated recurrent units (GRU) models to conduct multiple experiments and use root mean square error (RMSE) and residual predictive deviation (RPD) as evaluation metrics. Results The proposed method achieved an $$R^2$$ R 2 of 0.9989 on the test set data and improved by 3% over the traditional CNN. Multiple experiments were also conducted using RNN, LSTM, BiLSTM, and GRU models and evaluated their performance using RMSE, RPD, and other metrics. The results showed that the proposed method consistently outperformed these models. Conclusions This paper demonstrates the effectiveness of the proposed method for predicting the concentration of ofloxacin in solution based on Raman spectral data, in addition to discussing the advantages and limitations of the proposed method, and the study proposes a solution to the problem of deep learning methods for Raman spectral concentration prediction.

show abstract

Robust variable selection in the framework of classification with label noise and outliers: Applications to spectroscopic data in agri-food

Cited by 8 publications

References 56 publications

Modeling method for SSC prediction in pomelo using Vis-NIRS with wavelength selection and latent variable updating

Modeling method for SSC prediction in pomelo using Vis-NIRS with wavelength selection and latent variable updating

Noise simulation in classification with the noisemodel R package: Applications analyzing the impact of errors with chemical data

Raman spectroscopy-based prediction of ofloxacin concentration in solution using a novel loss function and an improved GA-CNN model

Contact Info

Product

Resources

About