AutoDEUQ: Automated Deep Ensemble with Uncertainty Quantification

Abstract

Deep neural networks are powerful predictors for a variety of tasks. However, they do not capture uncertainty directly. Using neural network ensembles to quantify uncertainty is competitive with approaches based on Bayesian neural networks while benefiting from better computational scalability. However, building ensembles of neural networks is a challenging task because, in addition to choosing the right neural architecture or hyperparameters for each member of the ensemble, there is an added cost of training each model. We propose Au-toDEUQ, an automated approach for generating an ensemble of deep neural networks. Our approach leverages joint neural architecture and hyperparameter search to generate ensembles. We use the law of total variance to decompose the predictive variance of deep ensembles into aleatoric (data) and epistemic (model) uncertainties. We show that AutoDEUQ outperforms probabilistic backpropagation, Monte Carlo dropout, deep ensemble, distribution-free ensembles, and hyper ensemble methods on a number of regression benchmarks.

Imprecise Bayesian Neural Networks

“…In [16], the authors pursue the same heuristic endeavor, but take an ensemble route. They too consider different BNNs, but instead of keeping them separate and use them to build a posterior credal set, they average them out.…”

“…Results. Following [16], for EBNN we posit that 1/k k j=1 σ 2 j captures the aleatoric uncertainty, and 1/(k − 1) k j=1 (µ j − µ ens ) 2 captures the epistemic uncertainty; we use these values as baselines. 10 We discuss the results for CIFAR10 presented in Table 3.…”

“…• by [38,Theorem 38], if P is convex, then (16) holds with an equality. Suppose now Ω is at most countable; in this case, we do not need any assumptions to make the Choquet integrals well defined, since we will not deal with density functions.…”

Toward Robust Uncertainty Estimation with Random Activation Functions

“…Uncertainty Quantification (UQ) is an active field of research and various methods have been proposed to efficiently estimate the uncertainty of machine learning models (see Abdar et al 2021 for an extensive overview). While most research focuses on Bayesian deep learning (Srivastava et al 2014;Blundell et al 2015;Sensoy, Kandemir, and Kaplan 2018;Fan et al 2020;Järvenpää, Vehtari, and Marttinen 2020;Charpentier, Zügner, and Günnemann 2020), deep ensemble methods, which benefit from the advantages of both deep learning and ensemble learning, have been recently leveraged for empirical uncertainty quantification (Egele et al 2021;Hoffmann, Fortmeier, and Elster 2021;Brown, Bhuiyan, and Talbert 2020;Althoff, Rodrigues, and Bazame 2021). Although Bayesian UQ methods have solid theoretical foundation, they often require significant changes to the training procedure and are computationally expensive compared to non-Bayesian techniques such as ensembles (Egele et al 2021;Rahaman and Thiery 2021;Lakshminarayanan, Pritzel, and Blundell 2017).…”

Real-Time Drift-Driving Control for an Autonomous Vehicle: Learning from Nonlinear Model Predictive Control via a Deep Neural Network

“…This enables the controller to avoid state spaces with high uncertainty owing to noisy exteroceptive input or insufficient model learning. UC(x) is defined as the total amount of uncertainty inherent in the model prediction, which can be expressed as the sum of aleatoric uncertainty AU(x) and epistemic uncertainty EU(x) as follows [29]:…”