A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglement

Abstract

Neural networks are ubiquitous in many tasks, but trusting their predictions is an open issue. Uncertainty quantification is required for many applications, and disentangled aleatoric and epistemic uncertainties are best. In this paper, we generalize methods to produce disentangled uncertainties to work with different uncertainty quantification methods, and evaluate their capability to produce disentangled uncertainties. Our results show that: there is an interaction between learning aleatoric and epistemic uncertainty, which is unexpected and violates assumptions on aleatoric uncertainty, some methods like Flipout produce zero epistemic uncertainty, aleatoric uncertainty is unreliable in the out-of-distribution setting, and Ensembles provide overall the best disentangling quality. We also explore the error produced by the number of samples hyper-parameter in the sampling softmax function, recommending N > 100 samples. We expect that our formulation and results help practitioners and researchers choose uncertainty methods and expand the use of disentangled uncertainties, as well as motivate additional research into this topic.

Addressing the Challenge of Ambiguous Data in Deep Learning

“…Our definition of ambiguity is also often confused or conflated with uncertainty. The literature mainly distinguishes between aleatoric and epistemic uncertainty [33,163], while acknowledging that it is difficult to distinguish between them in DL [1,83,144,178]. Aleatoric uncertainty is a statistical uncertainty that is inherent in the data and cannot be influenced by the model.…”