Can NLI Models Verify QA Systems' Predictions?

Abstract

To build robust question answering systems, we need the ability to verify whether answers to questions are truly correct, not just "good enough" in the context of imperfect QA datasets. We explore the use of natural language inference (NLI) as a way to achieve this goal, as NLI inherently requires the premise (document context) to contain all necessary information to support the hypothesis (proposed answer to the question). We leverage large pretrained models and recent prior datasets to construct powerful question converter and decontextualization modules, which can reformulate QA instances as premise-hypothesis pairs with very high reliability. Then, by combining standard NLI datasets with NLI examples automatically derived from QA training data, we can train NLI models to judge correctness of QA models' proposed answers. We show that our NLI approach can generally improve the confidence estimation of a QA model across different domains, evaluated in a selective QA setting. Careful manual analysis over the predictions of our NLI model shows that it can further identify cases where the QA model produces the right answer for the wrong reason, or where the answer cannot be verified as addressing all aspects of the question. Context: The first season of the fantasy comedy television series The Good Place, created by Michael Schur, aired … The series focuses on Eleanor Shellstrop (Kristen Bell) , a woman who wakes up in the afterlife and is introduced by Michael (Ted Danson) to a Heaven-like utopia he designed … Question: Who plays the bad guy in the Good Place? Answer: Ted Danson Premise: The series The Good Place focuses on Eleanor Shellstrop (Kristen Bell) , a woman who wakes up in the afterlife and is introduced by Michael (Ted Danson) to a Heaven-like utopia he designed. Hypothesis: Ted Danson plays the bad guy in The Good Place.

Mitigating Temporal Misalignment by Discarding Outdated Facts

“…Fact Duration Following suit with the QA evaluations above, we also evaluate fact duration prediction on SituatedQA. To generate fact-duration pairs, we use the annotated previous answer as of 2021, converting the question/answer pair into statement using an existing T5-based conversion model (Chen et al, 2021a). We then use distance between the 2021 and previous answer's start date as the fact's duration, d.…”

“…TimeQA (Chen et al, 2021b) is one such work that curates a dataset of 70 different temporally-dependent relations from Wikidata and uses handcrafted templates to convert into decontextualized QA pairs, where the question specifies a time period. To convert this dataset into factduration pairs (f, d), we first convert their QA pairs into a factual statements by removing the date and using a QA-to-statement conversion model (Chen et al, 2021a). We then determine the duration of each facts to be the length of time between the start date of one answer to the question and the next.…”

Answer Quality Aware Aggregation for Extractive QA Crowdsourcing

“…Answer Correctness (AC) QA models often lack the ability to verify the correctness of the predicted answer (Chen et al, 2021). One way to address this issue is to reformulate it to a textual entailment problem (Harabagiu and Hickl, 2006;Richardson et al, 2013;Chen et al, 2021) by viewing the answer context as the premise and the QA pair as the hypothesis. Then we use a natural language inference (NLI) system to verify whether the candidate answer proposed by crowd workers satisfies the entailment criterion.…”