Discrimination of Different Species of Dendrobium with an Electronic Nose Using Aggregated Conformal Predictor

Wang, You; Zhan, Wang; Diao, Junwei; Sun, Xiyang; Luo, Zhiyuan; Li, Guang

doi:10.3390/s19040964

“…CP models, or generally ML models, are widely used for molecular property predictions, including activity or toxicity 5,6,55 . Notably, the CP framework is based on the assumption that test and calibration data stem from the same distribution 10,11 .…”

Section: Discussionmentioning

confidence: 99%

Studying and Mitigating the Effects of Data Drifts on ML Model Performance at the Example of Chemical Toxicity Data

Morger¹,

Lomana²,

Norinder³

et al. 2021

Preprint

0

View full text Add to dashboard Cite

Machine learning models are widely applied to predict molecular properties or the biological activity of small molecules on a specific protein. Models can be integrated in a conformal prediction (CP) framework which adds a calibration step to estimate the confidence of the predictions. CP models present the advantage of ensuring a predefined error rate under the assumption that test and calibration set are exchangeable. In cases where the test data have drifted away from the descriptor space of the training data, or where assay setups have changed, this assumption might not be fulfilled and the models are not guaranteed to be valid. In this study, the performance of internally valid CP models when applied to either newer time-split data or to external data was evaluated. In detail, temporal data drifts were analysed based on twelve datasets from the ChEMBL database. In addition, discrepancies between models trained on publicly available data and applied to proprietary data for the liver toxicity and MNT in vivo endpoints were investigated. In most cases, a drastic decrease in the validity of the models was observed when applied to the time-split or external (holdout) test sets. To overcome the decrease in model validity, a strategy for updating the calibration set with data more similar to the holdout set was investigated. Updating the calibration set generally improved the validity, restoring it completely to its expected value in many cases. The restored validity is the first requisite for applying the CP models with confidence. However, the increased validity comes at the cost of a decrease in model efficiency, as more predictions are identified as inconclusive. This study presents a strategy to recalibrate CP models to mitigate the effects of data drifts. Updating the calibration sets without having to retrain the model has proven to be a useful approach to restore the validity of most models.

show abstract

“…CP models, or generally ML models, are widely used for molecular property predictions, including activity or toxicity 5,6,63 . Notably, the CP framework is based on the assumption that test and calibration data stem from the same distribution 10,11 .…”

Section: Update Calibration Strategy On Inhouse Datasetsmentioning

confidence: 99%

Studying and mitigating the effects of data drifts on ML model performance at the example of chemical toxicity data

Morger

¹

,

Lomana

²

,

Norinder

³

et al. 2022

Preprint

0

View full text Add to dashboard Cite

Machine learning models are widely applied to predict molecular properties or the biological activity of small molecules on a specific protein. Models can be integrated in a conformal prediction (CP) framework which adds a calibration step to estimate the confidence of the predictions. CP models present the advantage of ensuring a predefined error rate under the assumption that test and calibration set are exchangeable. In cases where the test data have drifted away from the descriptor space of the training data, or where assay setups have changed, this assumption might not be fulfilled and the models are not guaranteed to be valid. In this study, the performance of internally valid CP models when applied to either newer time-split data or to external data was evaluated. In detail, temporal data drifts were analysed based on twelve datasets from the ChEMBL database. In addition, discrepancies between models trained on publicly available data and applied to proprietary data for the liver toxicity and MNT in vivo endpoints were investigated. In most cases, a drastic decrease in the validity of the models was observed when applied to the time-split or external (holdout) test sets. To overcome the decrease in model validity, a strategy for updating the calibration set with data more similar to the holdout set was investigated. Updating the calibration set generally improved the validity, restoring it completely to its expected value in many cases. The restored validity is the first requisite for applying the CP models with confidence. However, the increased validity comes at the cost of a decrease in model efficiency, as more predictions are identified as inconclusive. This study presents a strategy to recalibrate CP models to mitigate the effects of data drifts. Updating the calibration sets without having to retrain the model has proven to be a useful approach to restore the validity of most models.

show abstract

“…Alternative herbal medicines are valuable in medical research and therapies [1], and their distinct treatment effects are associated with specific categories, which can be 1/24 arXiv:2102.03088v1 [cs.LG] 5 Feb 2021 discriminated with the different emanated volatile organic compounds (VOCs) [2]. However, the subtle differences in the herbal medicine appearance lead manual classification to rely heavily on the knowledge and experience of doctors and pharmacists [3]. For better patient treatment with correct alternative herbal medicine, it is worthwhile to develop cheap, effective and reproducible methods for the medicine classification [4].…”

Section: Introductionmentioning

confidence: 99%

“…To address the limitations of supervised learning with labelled data, Li et al [3,15,16] applied conformal prediction(CP) in effectively improving classification accuracy with augmented data. However, there are multiple limitations: firstly, the proposed onlinelearning method did not fully consider the real-world situations by using information that should be unavailable: for example, true labels were used by Wang et al [3] to validate the effectiveness of conformal prediction, however, the true labels should not have been accessible in the online-learning process. Secondly, the data used in previous research [3,15] assumed all the data to be homogeneous without considering environment changes or sensor drifts as the researchers used the same batch of data collected to validate the online learning protocols.…”

Section: Introductionmentioning

confidence: 99%

Boost AI Power: Data Augmentation Strategies With Unlabeled Data and Conformal Prediction, a Case in Alternative Herbal Medicine Discrimination With Electronic Nose

Liu

¹

,

Zhan

²

,

Wu

³

et al. 2021

Self Cite

View full text Add to dashboard Cite

Electronic nose proves its effectiveness in alternative herbal medicine classification, but due to the supervised learning nature, previous research relies on the labelled training data, which are time-costly and labor-intensive to collect. Considering the training data inadequacy in real-world applications, this study aims to improve classification accuracy via data augmentation strategies. We stimulated two scenarios to investigate the effectiveness of five data augmentation strategies under different training data inadequacy: in the noise-free scenario, different availability of unlabelled data were simulated, and in the noisy scenario, different levels of Gaussian noises and translational shifts were added to simulate sensor drifts. The augmentation strategies: noise-adding data augmentation, semi-supervised learning, classifier-based online learning, inductive conformal prediction (ICP) online learning and the novel ensemble ICP online learning proposed in this study, were compared against supervised learning baseline, with Linear Discriminant Analysis (LDA) and Support Vector Machine (SVM) as the classifiers. We found that at least one strategies significantly improved the classification accuracy with LDA(p ≤ 0.05) and showed non-decreasing classification accuracy with SVM in each tasks. Moreover, our novel strategy: ensemble ICP online learning outperformed the others by showing non-decreasing classification accuracy on all tasks and significant improvement on most tasks (25/36 tasks, p ≤ 0.05). This study provides a systematic analysis over augmentation strategies, and we provided users with recommended strategies under specific circumstances. Furthermore, our newly proposed strategy showed both effectiveness and robustness in boosting the classification model generalizability, which can also be further employed in other machine learning applications.

show abstract

Discrimination of Different Species of Dendrobium with an Electronic Nose Using Aggregated Conformal Predictor

Cited by 13 publications

References 53 publications

Studying and Mitigating the Effects of Data Drifts on ML Model Performance at the Example of Chemical Toxicity Data

Studying and Mitigating the Effects of Data Drifts on ML Model Performance at the Example of Chemical Toxicity Data

Studying and mitigating the effects of data drifts on ML model performance at the example of chemical toxicity data

Boost AI Power: Data Augmentation Strategies With Unlabeled Data and Conformal Prediction, a Case in Alternative Herbal Medicine Discrimination With Electronic Nose

Contact Info

Product

Resources

About