Will this Question be Answered? Question Filtering via Answer Model Distillation for Efficient Question Answering

Abstract

In this paper we propose a novel approach towards improving the efficiency of Question Answering (QA) systems by filtering out questions that will not be answered by them. This is based on an interesting new finding: the answer confidence scores of state-of-the-art QA systems can be approximated well by models solely using the input question text. This enables preemptive filtering of questions that are not answered by the system due to their answer confidence scores being lower than the system threshold. Specifically, we learn Transformer-based question models by distilling Transformer-based answering models. Our experiments on three popular QA datasets and one industrial QA benchmark demonstrate the ability of our question models to approximate the Precision/Recall curves of the target QA system well. These question models, when used as filters, can effectively trade off lower computation cost of QA systems for lower Recall, e.g., reducing computation by ∼60%, while only losing ∼3−4% of Recall.

Towards Improving Selective Prediction Ability of NLP Systems

“…The softmax probability assigned to class '1' by this calibrator is used as the confidence estimator for selective prediction. We refer to this approach as Calib C. We also train a transformer-based model for calibration (Calib T) that leverages the entire input text for this classifi-cation task instead of the syntactic features (Garg and Moschitti, 2021).…”

The Unreliability of Explanations in Few-Shot In-Context Learning

“…These confidence scores can be used in a "selective QA" setting (Kamath et al, 2020), where the model can abstain on a certain fraction of questions where it assigns low confidence to its answers. We use the area under coverage-accuracy curve(AUC) to evaluate how well a model is calibrated as in past literature (Kamath et al, 2020;Garg and Moschitti, 2021;Ye and Durrett, 2022). The curve plots the average accuracy with varying fractions (coverage) of questions being answered (examples in Figure 5).…”

Paragraph-based Transformer Pre-training for Multi-Sentence Inference

“…The performance of AS2 systems in practical applications is typically (Garg and Moschitti, 2021) measured using the Accuracy in providing correct answers for the questions (the percentage of correct responses provided by the system), also called the Precision-at-1 (P@1). In addition to P@1, we use Mean Average Precision (MAP) and Mean Reciprocal Recall (MRR) to evaluate the ranking produced of the set of candidates by the model.…”