If You Want to Go Far Go Together: Unsupervised Joint Candidate Evidence Retrieval for Multi-hop Question Answering

Abstract

Multi-hop reasoning requires aggregation and inference from multiple facts. To retrieve such facts, we propose a simple approach that retrieves and reranks set of evidence facts jointly. Our approach first generates unsupervised clusters of sentences as candidate evidence by accounting links between sentences and coverage with the given query. Then, a RoBERTa-based reranker is trained to bring the most representative evidence cluster to the top. We specifically emphasize on the importance of retrieving evidence jointly by showing several comparative analyses to other methods that retrieve and rerank evidence sentences individually. First, we introduce several attention-and embedding-based analyses, which indicate that jointly retrieving and reranking approaches can learn compositional knowledge required for multi-hop reasoning. Second, our experiments show that jointly retrieving candidate evidence leads to substantially higher evidence retrieval performance when fed to the same supervised reranker. In particular, our joint retrieval and then reranking approach achieves new state-of-the-art evidence retrieval performance on two multi-hop question answering (QA) datasets: 30.5 Recall@2 on QASC, and 67.6% F1 on MultiRC. When the evidence text from our joint retrieval approach is fed to a RoBERTa-based answer selection classifier, we achieve new state-ofthe-art QA performance on MultiRC and second best result on QASC.

RPA: Reasoning Path Augmentation in Iterative Retrieving for Multi-Hop QA

“…The similarity scores between tokens are effective in removing invalid information while keeping valid keywords, nevertheless, ignoring the contextual representation of the question. Additionally, Yadav et al (2021) propose the JointRR uses the same method of similarity score in AIR as the filter but with a RoBERTa re-ranker, presenting a better performance of retrieval. The answer classifier in AIR and JointRR is the same RoBERTa.…”

“…Justifications order. Only supervised methods are associated with the order of justifications in RP (Asai et al 2020;Li et al 2021), while the unsupervised methods ignore it (Yadav et al 2020(Yadav et al , 2021. PathRetriever adaptively scores each RP in the graph constructed with the Wikipedia hyperlinks and document structures to model the relationships.…”

A Survey on Multi-hop Question Answering and Generation

“…During training, heuristics are used to find the oracle query. • Yadav et al [171] retrieve k justification sentences using alignment technique similar to Yadav et al [170]. The question 𝑄, is concatenated with each retrieved justification 𝑞 𝑘 , and the token weights are assigned as: For each token 𝑡 in original question, if 𝑞 𝑘 contains 𝑡, weight for 𝑡 is 1, else it is 2.…”

“…• Zhang et al [181] pass the node (document) representations to a binary classifier for predicting its relevance and keep the 𝑘 highest scoring documents. • Yadav et al [171] use RoBERTa [92] for reranking trained to predict the F-1 score of evidence chain. • Zhang et al [179] and 20 concatenate paragraphs with the question and feed it to a BERT followed by a binary classifier and keep N(=3) paragraphs with the highest scores.…”

“…A Bi-directional attention layer on query and context [15]. These representations can be used to perform an independent binary classification for each candidate (for multiple-correct type questions such as MultiRC) [145,[169][170][171] or a multi-way classification for all candidates (for single-correct type questions such as OpenBookQA, WikiHop) [15,18,30,170,171]. De Cao et al [30] also use an ensemble of 5 models, each trained with a different weight initialisation.…”