Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks

Huseljic, Denis; Sick, Bernhard; Herde, Marek; Kottke, Daniel

doi:10.1109/icpr48806.2021.9412616

Cited by 9 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The effect is therefore expected to vanish for larger datasets. Huseljic et al postulated that high epistemic uncertainty entails high aleatoric uncertainty [ 61 ]. Thus, even for a KL weight of 0.1, the difference in the aleatoric uncertainties between the model trained on 100 samples compared to the ones trained on 200 and 400 samples remained visible even if the difference between the latter two equalized as expected.…”

Section: Discussionmentioning

confidence: 99%

Towards Reliable Parameter Extraction in MEMS Final Module Testing Using Bayesian Inference

Heringhaus

Zhang

Zimmermann

et al. 2022

Sensors

View full text Add to dashboard Cite

In micro-electro-mechanical systems (MEMS) testing high overall precision and reliability are essential. Due to the additional requirement of runtime efficiency, machine learning methods have been investigated in recent years. However, these methods are often associated with inherent challenges concerning uncertainty quantification and guarantees of reliability. The goal of this paper is therefore to present a new machine learning approach in MEMS testing based on Bayesian inference to determine whether the estimation is trustworthy. The overall predictive performance as well as the uncertainty quantification are evaluated with four methods: Bayesian neural network, mixture density network, probabilistic Bayesian neural network and BayesFlow. They are investigated under the variation in training set size, different additive noise levels, and an out-of-distribution condition, namely the variation in the damping factor of the MEMS device. Furthermore, epistemic and aleatoric uncertainties are evaluated and discussed to encourage thorough inspection of models before deployment striving for reliable and efficient parameter estimation during final module testing of MEMS devices. BayesFlow consistently outperformed the other methods in terms of the predictive performance. As the probabilistic Bayesian neural network enables the distinction between epistemic and aleatoric uncertainty, their share of the total uncertainty has been intensively studied.

show abstract

Section: Discussionmentioning

confidence: 99%

Towards Reliable Parameter Extraction in MEMS Final Module Testing Using Bayesian Inference

Heringhaus

Zhang

Zimmermann

et al. 2022

Sensors

View full text Add to dashboard Cite

show abstract

“…The evaluation on out-of-distribution (OOD) data is commonly done in classification (Huseljic, Sick, Herde, & Kottke, 2021) as it gives insights about the quality of uncertainty estimates of a model (e.g., probabilistic outputs or derived measures). The idea behind an OOD evaluation in an object detection setting, however, is fairly uncommon (Du, Wang, Cai, & Li, 2022).…”

Section: Setupmentioning

confidence: 99%

A Review of Uncertainty Calibration in Pretrained Object Detectors

Huseljic¹,

Herde²,

Muejde³

et al. 2022

Preprint

View full text Add to dashboard Cite

In the field of deep learning based computer vision, the development of deep object detection has led to unique paradigms (e.g., two-stage or set-based) and architectures (e.g., Faster-RCNN or DETR) which enable outstanding performance on challenging benchmark datasets. Despite this, the trained object detectors typically do not reliably assess uncertainty regarding their own knowledge, and the quality of their probabilistic predictions is usually poor. As these are often used to make subsequent decisions, such inaccurate probabilistic predictions must be avoided. In this work, we investigate the uncertainty calibration properties of different pretrained object detection architectures in a multi-class setting. We propose a framework to ensure a fair, unbiased, and repeatable evaluation and conduct detailed analyses assessing the calibration under distributional changes (e.g., distributional shift and application to out-of-distribution data). Furthermore, by investigating the influence of different detector paradigms, post-processing steps, and suitable choices of metrics, we deliver novel insights into why poor detector calibration emerges. Based on these insights, we are able to improve the calibration of a detector by simply finetuning its last layer.

show abstract

“…Prior Networks [31,32] use the aleatoric and epistemic uncertainty for OOD detection using a Dirichlet distribution. Another related method is proposed by Huseljic et al [23]. They utilize the properties of a Dirichlet-Categorical distribution and are able to measure and separate aleatoric and epistemic uncertainty.…”

Section: Out-of-distribution Detectionmentioning

confidence: 99%

“…Their model is then able to detect OOD samples while also allowing for a reliable estimation of the risk coming with a decision of the trained DNN. Both [23] and [28] require a separate set of (artificially generated) OOD samples to train OOD detectors.…”

Section: Out-of-distribution Detectionmentioning

confidence: 99%

See 1 more Smart Citation

Out-of-distribution Detection and Generation using Soft Brownian Offset Sampling and Autoencoders

Möller¹,

Botache²,

Huseljic³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks often suffer from overconfidence which can be partly remedied by improved outof-distribution detection. For this purpose, we propose a novel approach that allows for the generation of outof-distribution datasets based on a given in-distribution dataset. This new dataset can then be used to improve out-of-distribution detection for the given dataset and machine learning task at hand. The samples in this dataset are with respect to the feature space close to the in-distribution dataset and therefore realistic and plausible. Hence, this dataset can also be used to safeguard neural networks, i.e., to validate the generalization performance. Our approach first generates suitable representations of an in-distribution dataset using an autoencoder and then transforms them using our novel proposed Soft Brownian Offset method. After transformation, the decoder part of the autoencoder allows for the generation of these implicit out-of-distribution samples. This newly generated dataset then allows for mixing with other datasets and thus improved training of an out-of-distribution classifier, increasing its performance. Experimentally, we show that our approach is promising for time series using synthetic data. Using our new method, we also show in a quantitative case study that we can improve the out-of-distribution detection for the MNIST dataset. Finally, we provide another case study on the synthetic generation of out-ofdistribution trajectories, which can be used to validate trajectory prediction algorithms for automated driving.

show abstract

Separation of Aleatoric and Epistemic Uncertainty in Deterministic Deep Neural Networks

Cited by 9 publications

References 20 publications

Towards Reliable Parameter Extraction in MEMS Final Module Testing Using Bayesian Inference

Towards Reliable Parameter Extraction in MEMS Final Module Testing Using Bayesian Inference

A Review of Uncertainty Calibration in Pretrained Object Detectors

Out-of-distribution Detection and Generation using Soft Brownian Offset Sampling and Autoencoders

Contact Info

Product

Resources

About