Training Uncertainty-Aware Classifiers with Conformalized Deep Learning

Abstract

Deep neural networks are powerful tools to detect hidden patterns in data and leverage them to make predictions, but they are not designed to understand uncertainty and estimate reliable probabilities. In particular, they tend to be overconfident. We address this problem by developing a novel training algorithm that can lead to more dependable uncertainty estimates, without sacrificing predictive power. The idea is to mitigate overconfidence by minimizing a loss function, inspired by advances in conformal inference, that quantifies model uncertainty by carefully leveraging hold-out data. Experiments with synthetic and real data demonstrate this method leads to smaller conformal prediction sets with higher conditional coverage, after exact calibration with hold-out data, compared to state-of-the-art alternatives.

Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction

“…Through experimental results, including for modulation classification [11], [12], meta-XB was shown to outperform both conventional conformal prediction-based solutions and meta-learning conformal prediction schemes. Future work may integrate meta-learning with CP-aware training criteria [41], [42], or with stochastic set predictors.…”

“…CP-aware loss. [41] and [42] proposed CP-aware loss functions to enhance the efficiency or per-input validity (3) of VB-CP. The drawback of these solutions is that they require a large amount of data samples, i.e., N τ 1, unlike the meta-learning methods studied here.…”