Uncertainty-Aware Reliable Text Classification

Abstract

Deep neural networks have significantly contributed to the success in predictive accuracy for classification tasks. However, they tend to make over-confident predictions in real-world settings, where domain shifting and out-of-distribution (OOD) examples exist. Most research on uncertainty estimation focuses on computer vision because it provides visual validation on uncertainty quality. However, few have been presented in the natural language process domain. Unlike Bayesian methods that indirectly infer uncertainty through weight uncertainties, current evidential uncertainty-based methods explicitly model the uncertainty of class probabilities through subjective opinions. They further consider inherent uncertainty in data with different root causes, vacuity (i.e., uncertainty due to a lack of evidence) and dissonance (i.e., uncertainty due to conflicting evidence). In our paper, we firstly apply evidential uncertainty in OOD detection for text classification tasks. We propose an inexpensive framework that adopts both auxiliary outliers and pseudo off-manifold samples to train the model with prior knowledge of a certain class, which has high vacuity for OOD samples. Extensive empirical experiments demonstrate that our model based on evidential uncertainty outperforms other counterparts for detecting OOD examples. Our approach can be easily deployed to traditional recurrent neural networks and fine-tuned pre-trained transformers. CCS CONCEPTS• Computing methodologies → Natural language processing; Neural networks.

ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence

“…Due to limited time and computational resources, we did not conduct more experiments to explore various hyperparameters that could affect finetuning results, such as vocabulary size and pretraining epochs, to name a few. Future work should analyze how to optimize ConfliBERT, expand Con-fliBERT to multi-lingual settings, and apply Con-fliBERT to more challenging tasks such as understanding, inference, question answering, uncertainty qualification Hu and Khan, 2021), and few-/ zero-shot tasks to speed up the study of NLP application for the political science community.…”

School Mathematics Textbooks in China

“…Benefit from self-training, previous methods make a remarkable success on a series of instance-level classification tasks, such as image classification (Zhou et al 2021;Wang et al 2022a;Liu et al 2021) and text classification (Meng et al 2020;Mukherjee and Awadallah 2020;Yu et al 2021;Hu and Khan 2021;Tsai, Lin, and Fu 2022;Kim, Son, and Han 2022). In contrast to instance-level classification, we observe that there are two challenges in applying standard self-training to NSL.…”

Domain-Invariant Feature Distillation for Cross-Domain Sentiment Classification

“…Confidence [88,112,114,115]; Entropy [76,113]; TS [113]; KL [103]; Mahalanobis [88,89]; Evidence [117]; MCD [116]; Ensemble [116]; Gaussian [113].…”

“…Accuracy [112,114,116]; [76,103,113]; F1, [89,115,116]; ECE [114]; AUROC [88,113,117]; AUPR [89,113,117]; FAR95 [88,112,113]; AUCRCC [89], FPR90 [117].…”

“…They further use Mahalanobis distance and Gaussian distribution in the penultimate layer to accurately detect OOD instances. Hu et al [117] exploit uncertainty in OOD detection for text classification tasks using evidential neural networks based on Dirichlet distribution. They propose a framework that adopts auxiliary outliers and pseudo off-manifold samples to train the model with prior knowledge of a certain class, which has high vacuity for OOD samples.…”