Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering

Abstract

We propose an unsupervised strategy for the selection of justification sentences for multihop question answering (QA) that (a) maximizes the relevance of the selected sentences, (b) minimizes the overlap between the selected facts, and (c) maximizes the coverage of both question and answer. This unsupervised sentence selection method can be coupled with any supervised QA approach. We show that the sentences selected by our method improve the performance of a state-of-the-art supervised QA model on two multi-hop QA datasets: AI2's Reasoning Challenge (ARC) and Multi-Sentence Reading Comprehension (MultiRC). We obtain new state-of-the-art performance on both datasets among approaches that do not use external resources for training the QA system: 56.82% F1 on ARC (41.24% on Challenge and 64.49% on Easy) and 26.1% EM0 on MultiRC. Our justification sentences have higher quality than the justifications selected by a strong information retrieval baseline, e.g., by 5.4% F1 in MultiRC. We also show that our unsupervised selection of justification sentences is more stable across domains than a state-of-the-art supervised sentence selection method.

Unsupervised Alignment-based Iterative Evidence Retrieval for Multi-hop Question Answering

“…Our work falls under the revitalized direction that focuses on the interpretability of QA systems, where the machine's inference process is explained to the end user in natural language evidence text (Qi et al, 2019;Yang et al, 2018;Wang et al, 2019b;Yadav et al, 2019b;Bauer et al, 2018). Several…”

“…As shown by (Khot et al, 2019a;Qi et al, 2019), and our table 2, these techniques do not work well for complex multi-hop questions, which require knowledge aggregation from multiple related justifications. Some unsupervised methods extract groups of justification sentences Yadav et al, 2019b) but these methods are exponentially expensive in the retrieval step. Contrary to all of these, AIR proposes a simpler and more efficient method for chaining justification sentences.…”

“…While several neural methods have achieved state-of-theart results on these datasets (Devlin et al, 2019;Liu et al, 2019;Yang et al, 2019), we argue that many of these directions lack a human-understandable explanation of their inference process, which is necessary to transition these approaches into realworld applications. This is especially critical for multi-hop, multiple choice QA (MCQA) where: (a) the answer text may not come from an actual knowledge base passage, and (b) reasoning is required to link the candidate answers to the given question (Yadav et al, 2019b). Figure 1 shows one such multi-hop example from a MCQA dataset.…”

Multi-hop Inference for Question-driven Summarization

“…Iida et al (2019) and Nakatsuji and Okui (2020) incorporate some background knowledge into Seq2Seq model for why questions and conclusion-centric questions. Some latest works (Feldman and El-Yaniv, 2019;Yadav et al, 2019;Nishida et al, 2019a) attempt to provide evidence or justifications for humanunderstandable explanation of the multi-hop inference process in factoid QA, where the inferred evidences are only treated as the middle steps for finding the answer. However, in non-factoid QA, the intermediate output is also important to form a complete answer, which requires a bridge between the multi-hop inference and summarization.…”

Do Transformers Dream of Inference, or Can Pretrained Generative Models Learn Implicit Inferential Rules?

“…The emergence of large pretrained language models (LM) (Devlin et al, 2019;Liu et al, 2019) yielded significant progress in question answering (QA), including complex QA tasks that require multihop reasoning (Banerjee et al, 2019;Asai et al, 2019;Yadav et al, 2019). Most of these stateof-the-art (SOTA) approaches address multi-hop reasoning tasks in a discriminative manner: they take the question, the candidate answer, and all the context available as the input, and produce a single score indicating the likelihood of the answer as justified by the provided context (an example is shown in Figure 1).…”