Uncertainty Estimation for Language Reward Models

Abstract

Language models can learn a range of capabilities from unsupervised training on text corpora. However, to solve a particular problem (such as text summarization) it is typically necessary to fine-tune them on a task-specific dataset. It is often easier for humans to choose between options than to provide labeled data, and prior work has achieved state-of-the-art performance by training a reward model from such preference comparisons. However, collecting a large preference comparison dataset is still expensive-and the learned reward models are unreliable out-of-distribution. We seek to address these problems via uncertainty estimation, which can improve sample efficiency and robustness using active learning and risk-averse reinforcement learning (RL). Specifically, we use bootstrap aggregating (bagging) to train an ensemble of reward models differing in the initialization of their final layer. Ensembles have proved successful in prior applications of active learning [9, 3], but we find that in our setting ensemble active learning does not outperform random sampling. Further experiments show that while the aggregate predictions are well-calibrated, the ensemble's estimated epistemic uncertainty is only weakly correlated with model error. We suspect this is because the ensemble members are fine-tuned from a single model and so are similar to one another. This suggests current pre-training methods will need to be modified to support uncertainty estimation, e.g. by training multiple language models. * Work conducted during an internship at DeepMind. Preprint. Under review.

Comparing and Improving Active Learning Uncertainty Measures for Transformer Models by Discarding Outliers

“…Still, it can be safely stated already that the two Softmax Ensemble techniques (KLD and VE) perform far worse than any other technique. This confirms the work of Gleave and Irving (2022), where the usefulness of softmax ensemble methods for Transformer models was investigated, with the same conclusion as ours: ensemble methods perform far worse than even random sampling for Transformer models.…”

“…The aforementioned Uncertainty-measures can directly be used in AL strategies to sort the pool of unlabeled samples to select exactly those samples for labeling that have the lowest confidence/highest uncertainty. As repeatedly reported by others Karamcheti et al (2021); Gleave and Irving (2022); We propose three easily implementable methods on improving uncertainty-based AL strategies by preventing potentially harmful outliers from being selected for labeling. An uncertainty-based AL strategy always selects those samples for labeling first, where the uncertainty is the highest.…”

“…But as has been mentioned in the past by other researchers Lakshminarayanan et al (2017); Weiss and Tonella (2022); Gleave and Irving (2022); Sankararaman et al (2022); D'Arcy and Downey (2022), the training objective for NNs is purely to maximize the value of the correct output neuron, not to create a true confidence-probability. An inherent limitation of the softmax function is its inability to have -in the theoretical case -zero confidence in its prediction, as the sum of all possible outcomes always equals 1.…”

“…Despite successful application in a variety of domains (Gonsior et al, 2020;Gal et al, 2017;Lowell et al, 2019), AL fails to work for very deep NN such as Transformerencoder models, rarely beating pure random sampling. The common explanation (Karamcheti et al, 2021;Gleave and Irving, 2022;Sankararaman et al, 2022;D'Arcy and Downey, 2022) is that AL methods favor hard-to-learn samples, often simply called outliers, which therefore neglects the potential benefits gained from AL. Another potential explanation -to the best of our knowledge not yet covered in the AL literature -could be the calculation of the uncertainty of the NN.…”

Navigating the Grey Area: How Expressions of Uncertainty and Overconfidence Affect Language Models

“…While scholars have studied model uncertainty, prior work has focused on more accurately extracting model confidence (Kuhn et al, 2023;Sun et al, 2022;Gleave and Irving, 2022), measuring (Kwiatkowski et al, 2019b;Radford et al, 2019;Liang et al, 2023) and improving model calibration 2022) teach models to be linguistically calibrated when answering math questions.…”