Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty

Senge, Robin; Bösner, Stefan; Dembczyński, Krzysztof; Haasenritter, Jörg; Hirsch, Oliver; Donner-Banzhoff, Norbert; Hüllermeier, Eyke

doi:10.1016/j.ins.2013.07.030

Cited by 114 publications

(102 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, this distinction has also received attention in machine learning, where the "agent" is a learning algorithm [18]. In particular, a distinction between aleatoric and epistemic uncertainty has been advocated in the literature on deep learning [6], where the limited awareness of neural networks of their own competence has been demonstrated quite nicely.…”

Section: Introductionmentioning

confidence: 99%

Aleatoric and Epistemic Uncertainty with Random Forests

Shaker

Hüllermeier

2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Due to the steadily increasing relevance of machine learning for practical applications, many of which are coming with safety requirements, the notion of uncertainty has received increasing attention in machine learning research in the last couple of years. In particular, the idea of distinguishing between two important types of uncertainty, often refereed to as aleatoric and epistemic, has recently been studied in the setting of supervised learning. In this paper, we propose to quantify these uncertainties with random forests. More specifically, we show how two general approaches for measuring the learner's aleatoric and epistemic uncertainty in a prediction can be instantiated with decision trees and random forests as learning algorithms in a classification setting. In this regard, we also compare random forests with deep neural networks, which have been used for a similar purpose.

show abstract

Section: Introductionmentioning

confidence: 99%

Aleatoric and Epistemic Uncertainty with Random Forests

Shaker

Hüllermeier

2020

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

show abstract

“…According to [95], one can distinguish aleatoric and epistemic uncertainty in the classifier. In the context of LVQ, aleatoric uncertainty is due to randomness in data generating scheme whereas epimistic uncertainty is related to the lack of knowledge.…”

Section: Reject or Classify -Secure Classificationmentioning

confidence: 99%

Can Learning Vector Quantization be an Alternative to SVM and Deep Learning? - Recent Trends and Advanced Variants of Learning Vector Quantization for Classification Learning

Villmann

Bohnsack

Kaden

2016

Journal of Artificial Intelligence and Soft Computing Research

View full text Add to dashboard Cite

Learning vector quantization (LVQ) is one of the most powerful approaches for prototype based classification of vector data, intuitively introduced by Kohonen. The prototype adaptation scheme relies on its attraction and repulsion during the learning providing an easy geometric interpretability of the learning as well as of the classification decision scheme. Although deep learning architectures and support vector classifiers frequently achieve comparable or even better results, LVQ models are smart alternatives with low complexity and computational costs making them attractive for many industrial applications like intelligent sensor systems or advanced driver assistance systems.Nowadays, the mathematical theory developed for LVQ delivers sufficient justification of the algorithm making it an appealing alternative to other approaches like support vector machines and deep learning techniques.This review article reports current developments and extensions of LVQ starting from the generalized LVQ (GLVQ), which is known as the most powerful cost function based realization of the original LVQ. The cost function minimized in GLVQ is an soft-approximation of the standard classification error allowing gradient descent learning techniques. The GLVQ variants considered in this contribution, cover many aspects like bordersensitive learning, application of non-Euclidean metrics like kernel distances or divergences, relevance learning as well as optimization of advanced statistical classification quality measures beyond the accuracy including sensitivity and specificity or area under the ROC-curve.According to these topics, the paper highlights the basic motivation for these variants and extensions together with the mathematical prerequisites and treatments for integration into the standard GLVQ scheme and compares them to other machine learning approaches. For detailed description and mathematical theory behind all, the reader is referred to the respective original articles.Thus, the intention of the paper is to provide a comprehensive overview of the stateof-the-art serving as a starting point to search for an appropriate LVQ variant in case of a given specific classification problem as well as a reference to recently developed variants and improvements of the basic GLVQ scheme.

show abstract

“…The term relates to epistemic uncertainty (as opposed to aleatoric uncertainty) [10] and is due to our mistreatment of the evidence X with respect to the ideal model.…”

Section: Ideal Scores Q and The Decomposition L = El + Ilmentioning

confidence: 99%

“…This type of uncertainty is called aleatoric [10] so the loss could also be called aleatoric loss. It is the loss of the optimal model and equals zero only if the attributes of the instance X provide enough information to uniquely determine the right label Y (with probability 1).…”

Section: Ideal Scores Q and The Decomposition L = El + Ilmentioning

confidence: 99%

Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration

Kull

Flach

2015

Lecture Notes in Computer Science

View full text Add to dashboard Cite

General rightsThis document is made available in accordance with publisher policies. Please cite only the published version using the reference above. Abstract. There are several reasons to evaluate a multi-class classifier on other measures than just error rate. Perhaps most importantly, there can be uncertainty about the exact context of classifier deployment, requiring the classifier to perform well with respect to a variety of contexts. This is commonly achieved by creating a scoring classifier which outputs posterior class probability estimates. Proper scoring rules are loss evaluation measures of scoring classifiers which are minimised at the true posterior probabilities. The well-known decomposition of the proper scoring rules into calibration loss and refinement loss has facilitated the development of methods to reduce these losses, thus leading to better classifiers. We propose multiple novel decompositions including one with four terms: adjustment loss, post-adjustment calibration loss, grouping loss and irreducible loss. The separation of adjustment loss from calibration loss requires extra assumptions which we prove to be satisfied for the most frequently used proper scoring rules: Brier score and log-loss. We propose algorithms to perform adjustment as a simpler alternative to calibration.

show abstract

Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty

Cited by 114 publications

References 16 publications

Aleatoric and Epistemic Uncertainty with Random Forests

Aleatoric and Epistemic Uncertainty with Random Forests

Can Learning Vector Quantization be an Alternative to SVM and Deep Learning? - Recent Trends and Advanced Variants of Learning Vector Quantization for Classification Learning

Novel Decompositions of Proper Scoring Rules for Classification: Score Adjustment as Precursor to Calibration

Contact Info

Product

Resources

About