The MBPEP: a deep ensemble pruning algorithm providing high quality uncertainty prediction

Abstract

Machine learning algorithms have been effectively applied into various real world tasks. However, it is difficult to provide high-quality machine learning solutions to accommodate an unknown distribution of input datasets; this difficulty is called the uncertainty prediction problems. In this paper, a margin-based Pareto deep ensemble pruning (MBPEP) model is proposed. It achieves the high-quality uncertainty estimation with a small value of the prediction interval width (MPIW) and a high confidence of prediction interval coverage probability (PICP) by using deep ensemble networks. In addition to these networks, unique loss functions are proposed, and these functions make the sub-learners available for standard gradient descent learning. Furthermore, the margin criterion fine-tuning-based Pareto pruning method is introduced to optimize the ensembles. Several experiments including predicting uncertainties of classification and regression are conducted to analyze the performance of MBPEP. The experimental results show that MBPEP achieves a small interval width and a low learning error with an optimal number of ensembles. For the real-world problems, MBPEP performs well on input datasets with unknown distributions datasets incomings and improves learning performance on a multi task problem when compared to that of each single model.

A review of uncertainty quantification in deep learning: Techniques, applications and challenges

“…Liu et al [194] presented a Bayesian nonparametric ensemble method which enhanced an ensemble model that augmented model's distribution functions using Bayesian nonparametric machinery and prediction mechanism. Hu et al [195] proposed a model called margin-based Pareto deep ensemble pruning utilizing deep ensemble network that yielded competitive uncertainty estimation with elevated confidence of prediction interval coverage probability and a small value of the prediction interval width. In another study, the researchers [196] exploited the challenges associated with attaining uncertainty estimations for structured predictions job and presented baselines for sequence-level out-of-domain input detection, sequence-level prediction rejection and token-level error detection utilizing ensembles.…”

“…Deep ensemble, is another powerful method used to measure uncertainty and has been extensively applied in many real-world applications [195]. To achieve good learning results, the data distributions in testing datasets should be as close as the training datasets.…”

PID-GAN: A GAN Framework based on a Physics-informed Discriminator for Uncertainty Quantification with Physics

“…Uncertainty quantification (UQ) is an important end-goal in several real-world scientific applications where it is vital to produce distributions of output predictions as opposed to point estimates, allowing for meaningful analyses of the confidence in our predictions. In the context of deep learning, a number of techniques have been developed for UQ, including the use of Bayesian approximations [8,18,19] and ensemble-based methods [6,13,14,24]. A simple approach for performing UQ given a trained DL model is to apply Monte Carlo (MC)-Dropout on the DL weights during testing [4].…”

AVMSN: An Audio-Visual Two Stream Crowd Counting Framework Under Low-Quality Conditions

“…CNN-based counting. With the development of deep learning [11][12][13][14], the CNN-based methods [15][16][17][18] can transform the highly congested images into the densityestimation problem. The point-level labeling is adopted to annotate the objects, and the density map is generated following the Gaussian filter [17].…”