Benchmarking Uncertainty Quantification on Biosignal Classification Tasks under Dataset Shift

Abstract

A biosignal is a signal that can be continuously measured from human bodies, such as respiratory sounds, heart activity (ECG), brain waves (EEG), etc, based on which, machine learning models have been developed with very promising performance for automatic disease detection and health status monitoring. However, dataset shift, i.e., data distribution of inference varies from the distribution of the training, is not uncommon for real biosignal-based applications. To improve the robustness, probabilistic models with uncertainty qualification are adapted to capture how reliable a prediction is. Yet, assessing the quality of the estimated uncertainty remains a challenge. In this work, we propose a framework to evaluate the capability of the estimated uncertainty in capturing different types of biosignal dataset shifts with various degrees. In particular, we use three classification tasks based on respiratory sounds and electrocardiography signals to benchmark five representative uncertainty qualification methods. Extensive experiments show that, although Ensemble and Bayesian models could provide relatively better uncertainty estimations under dataset shifts, all tested models fail to meet the promise in trustworthy prediction and model calibration. Our work paves the way for a comprehensive evaluation for any newly developed biosignal classifiers.

Uncertainty-aware Health Diagnostics via Class-balanced Evidential Deep Learning

“…Thanks to uncertainty measurements, these models have shown superior performance and improved robustness in realistic deployment settings. Moving beyond image models, Xia et al [10], [45], [52] have benchmarked uncertainty estimation methods on various data modalities, i.e., respiratory sounds, heart activity, brain waves, etc. Although uncertainty-aware modeling has been explored in various applications, the existing focus has mainly been on simple Softmax-based approaches that can be overly confident, or MCDP and Ensembles that require substantial com-putational power and are challenging to deploy in realworld scenarios.…”