Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization

Abstract

Natural Language Inference (NLI) has garnered significant attention in recent years; however, the promise of applying NLI breakthroughs to other downstream NLP tasks has remained unfulfilled. In this work, we use the multiple-choice reading comprehension (MCRC) and checking factual correctness of textual summarization (CFCS) tasks to investigate potential reasons for this. Our findings show that: (1) the relatively shorter length of premises in traditional NLI datasets is the primary challenge prohibiting usage in downstream applications (which do better with longer contexts); (2) this challenge can be addressed by automatically converting resource-rich reading comprehension datasets into longer-premise NLI datasets; and (3) models trained on the converted, longer-premise datasets outperform those trained using shortpremise traditional NLI datasets on downstream tasks primarily due to the difference in premise lengths.

Can NLI Models Verify QA Systems’ Predictions?

“…The NLI models appear to be complementary to the QA model, improving performance even on out-of-domain data. We also see that our our NQ-NLI+MNLI+QA outperforms Mishra et al (2021)+QA by a large margin. By inspecting the performance breakdown in Appendix C, we see the gap is mainly on SQuAD2.0 and SQuADadv.…”

“…Moreover, the focus of other recent work in this space has been on transforming QA datasets into NLI datasets, which is a different end. Demszky et al (2018) and Mishra et al (2021) argue that QA datasets feature more diverse reasoning and can lead to stronger NLI models, particularly those better suited to strong contexts, but less attention has been paid to whether this agrees with classic definitions of entailment (Dagan et al, 2005) or short-context NLI settings (Williams et al, 2018).…”

“…First, it makes transferring models from the existing datasets challenging. Second, performing inference over longer forms of text requires a multitude of additional reasoning skills like coreference resolution, event detection, and abduction (Mishra et al, 2021). Finally, the presence of extraneous information makes it harder to evaluate the entailment model's judgments for correctness; in the extreme, we might have to judge whether a fact about an entity is true based on its entire Wikipedia article, which is impractical.…”

“…QA-enhanced NLI Model As there might exist different reasoning patterns in the QA datasets which are not covered by the MNLI model (Mishra et al, 2021), we study whether NLI pairs generated from QA datasets can be used jointly with the MNLI data to improve the performance of an NLI model. To do so, we run the QA instances in the NQ training set through our QA-to-NLI conversion pipeline, resulting in a dataset we call NQ-NLI, containing (premise, hypothesis) pairs from NQ with binary labels.…”

“…Our work is most relevant to Mishra et al (2021); they also learn an NLI model using examples generated from QA datasets. Our work differs from theirs in a few chief ways.…”