Learning to Transform, Combine, and Reason in Open-Domain Question Answering

Abstract

Users seek direct answers to complex questions from large open-domain knowledge sources like the Web. Open-domain question answering has become a critical task to be solved for building systems that help address users' complex information needs. Most open-domain question answering systems use a search engine to retrieve a set of candidate documents, select one or a few of them as context, and then apply reading comprehension models to extract answers. Some questions, however, require taking a broader context into account, e.g., by considering low-ranked documents that are not immediately relevant, combining information from multiple documents, and reasoning over multiple facts from these documents to infer the answer. In this paper, we propose a deep learning model based on the Transformer architecture that is able to efficiently operate over a larger set of candidate documents by effectively combining the evidence from these documents during multiple steps of reasoning, while it is robust against noise from low-ranked non-relevant documents included in the set.For example, in Figure 1, in order to infer the correct answer to the question: "Who is the Spanish artist, sculptor and draughtsman famous for co-founding the Cubist movement?" given the top-ranked document, a reading comprehension system most likely will extract "Georges Braque" as the answer, which is not the correct answer. In this example, in order to infer the correct answer, one has to go down the ranked list, gather and encode facts, even those that are not immediately relevant to the question, like "Malaga is a city in Spain," which can be inferred from a document at rank 66, and then in a multi-step reasoning process, infer some new facts, including "Picasso was a Spanish artist" given documents at ranks 12 and 66, and "Picasso, who was a Spanish artist, co-founded the Cubist" given the previously inferred fact and the document ranked third. In this example, and in general in many cases in open-domain question answering, a piece of information in a low-ranked document that is not immediately relevant to the question, may be useful to fill in the blanks and complete information extracted from the top relevant documents and eventually support inferring the correct answer.In this paper, we propose TraCRNet (pronounced Tracker Net) to improve opendomain question answering by explicitly operating on a larger set of candidate documents during the whole question answering process and learning how to aggregate and This is an extended abstract of Dehghani et al. [1].

Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering

“…In this work we propose an unsupervised algorithm for the selection of multi-hop justifications from unstructured knowledge bases (KB). Unlike other supervised selection methods (Dehghani et al, 2019;Bao et al, 2016;Wang et al, 2018b,a;Tran and Niedereée, 2018;Trivedi et al, 2019), our approach does not require any training data for justification selection. Unlike approaches that rely on structured KBs, which are expensive to create, (Khashabi et al, 2016;Khot et al, 2017;Khashabi et al, 2018b;Cui et al, 2017;Bao et al, 2016), our method operates over KBs of only unstructured texts.…”

Answering Complex Questions by Joining Multi-Document Evidence with Quasi Knowledge Graphs

“…Nevertheless, we compared with, and outperformed, the state-of-the-art system DrQA [12], which can both select relevant documents and extract answers from them. Traditional fact-centric QA over text, and multidocument reading comprehension are recently emerging as a joint topic referred to as open-domain question answering [16,42].…”

Token-level Dynamic Self-Attention Network for Multi-Passage Reading Comprehension

“…Some of the above datasets provide additional meta-data, we do not use this additional information in our experiments. We observe that those low-ranked passages play a critical role in improving the accuracy, thus we remain all supporting passages as the inputs of our (Chen et al, 2017), R 3 (Wang et al, 2018a), TraCRNet (Dehghani et al, 2019a), Shared-Norm (Clark and Gardner, 2018), HAS-QA (Pang et al, 2019). Human performance is referenced from the dataset paper.…”

“…In Wang et al (2018b), cross-passage answer verification is definitely proposed, in which all the word embeddings in a passage are summed through attention mechanism to represent an answer candidate, and then each answer candidate attends to other candidates to collect supportive information. In Dehghani et al (2019a), multihop reasoning is implemented by a Universal Transformer (Dehghani et al, 2019b) which is mainly based on Multi-head Self-attention (Vaswani et al, 2017) and a transition function.…”