Posterior Calibrated Training on Sentence Classification Tasks

Abstract

Most classification models work by first predicting a posterior probability distribution over all classes and then selecting that class with the largest estimated probability. In many settings however, the quality of posterior probability itself (e.g., 65% chance having diabetes), gives more reliable information than the final predicted class alone. When these methods are shown to be poorly calibrated, most fixes to date have relied on posterior calibration, which rescales the predicted probabilities but often has little impact on final classifications. Here we propose an end-to-end training procedure called posterior calibrated (PosCal) training that directly optimizes the objective while minimizing the difference between the predicted and empirical posterior probabilities. We show that PosCal not only helps reduce the calibration error but also improve task performance by penalizing drops in performance of both objectives. Our PosCal achieves about 2.5% of task performance gain and 16.1% of calibration error reduction on GLUE (Wang et al., 2018) compared to the baseline. We achieved the comparable task performance with 13.2% calibration error reduction on xSLUE (Kang and Hovy, 2019), but not outperforming the two-stage calibration baseline. PosCal training can be easily extendable to any types of classification tasks as a form of regularization term. Also, PosCal has the advantage that it incrementally tracks needed statistics for the calibration objective during the training process, making efficient use of large training sets 1 .

Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models

“…Since NCE training requires more computation (because of the noise ratio), we have tried finetuning the baseline with more steps, but we find that gives worse ECE and very little or no improvement on accuracy. We compare EBM training with three strong baselines for calibration: posterior calibrated training (PosCal) (Jung et al, 2020), temperature scaling (T-Scal) (Guo et al, 2017), and scaling-binning calibrator (Scal-bin) (Kumar et al, 2019). For PosCal and Scal-bin, we use the published code.…”

“…ECE first partitions all predictions into B equally-spaced bins by its confidence. Following Jung et al (2020); Grathwohl et al (2019), we set B " 20, which means the width of each bin is 0.05. For example, the first bin contains all predictions that have confidence in the range of r0, 0.05q.…”

“…available. To overcome this constraint, Jung et al (2020) uses a penalty term to encourage better calibration during training.…”

Platt-Bin: Efficient Posterior Calibrated Training for NLP Classifiers

“…This is because the number of classes are exponentially large and estimation of every posterior density or marginal posterior density is not possible. Previous works such as (Jung et al, 2020;Nguyen and O'Connor, 2015) propose to use the downstream task with small number of classes to perform calibration and estimation of the calibration error. In structured prediction models, calibration is also important for the generation of the structured outputs as the decoding algorithm relies on the posterior estimates to efficiently search through the space of sequences.…”

Clinical Natural Language Processing for Radiation Oncology: A Review and Practical Primer

“…Predicted probabilities of direct models are usually calibrated to present refined confidence scores, because no models are perfect, and even within high probability predicted labels, there could be errors. [23][24][25][26] An important distinction in machine learning algorithms, including deep learning algorithms, is whether they are supervised or unsupervised. Supervised learning algorithms predict target variables, which may be either continuous or categorical, such as prognosis, diagnosis, or toxicity.…”