Hierarchical Graph Network for Multi-hop Question Answering

Fang, Yuwei; Sun, Siqi; Gan, Zhe; Pillai, Rohit; Wang, Shuohang; Liu, Jingjing

doi:10.18653/v1/2020.emnlp-main.710

Cited by 100 publications

(108 citation statements)

References 39 publications

Supporting

Mentioning

107

Contrasting

Order By: Relevance

“…Additionally, it relies on a Graph Neural Network (GNN) to answer the questions. Hierarchical Graph Network (HGN) model (Fang et al, 2020) builds a hierarchical graph with three levels: entities, sentences and paragraphs to allow for joint reasoning. De-compRC (Min et al, 2019b) takes a completely different approach of learning to decompose the question (using additional annotations) and then answer the decomposed questions using a standard single-hop RC system.…”

Section: Related Workmentioning

confidence: 99%

A Simple Yet Strong Pipeline for HotpotQA

Groeneveld¹,

Khot²,

Sabharwal³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

State-of-the-art models for multi-hop question answering typically augment large-scale language models like BERT with additional, intuitively useful capabilities such as named entity recognition, graph-based reasoning, and question decomposition.However, does their strong performance on popular multihop datasets really justify this added design complexity? Our results suggest that the answer may be no, because even our simple pipeline based on BERT, named QUARK, performs surprisingly well. Specifically, on Hot-potQA, QUARK outperforms these models on both question answering and support identification (and achieves performance very close to a RoBERTa model). Our pipeline has three steps: 1) use BERT to identify potentially relevant sentences independently of each other; 2) feed the set of selected sentences as context into a standard BERT span prediction model to choose an answer; and 3) use the sentence selection model, now with the chosen answer, to produce supporting sentences. The strong performance of QUARK resurfaces the importance of carefully exploring simple model designs before using popular benchmarks to justify the value of complex techniques.

show abstract

Section: Related Workmentioning

confidence: 99%

A Simple Yet Strong Pipeline for HotpotQA

Groeneveld¹,

Khot²,

Sabharwal³

2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

show abstract

“…We achieved a test F1 of 79.34 and Exact Match (EM) of 66.33. Our approach is competitive with state-of-the-art systems SAE (Tu et al, 2020) and HGN (Fang et al, 2019), which both (unlike us) learn from strong, supporting-fact supervision about which sentences are relevant to the question.…”

Section: Results On Question Answeringmentioning

confidence: 98%

“…First, QA models that use decompositions outperform a strong RoBERTa baseline Min et al, 2019a) by 3.1 points in F1 on the original dev set, 10 points on the out-of-domain dev set from Min et al (2019b), and 11 points on the multi-hop dev set from Jiang and Bansal (2019a). Our method is competitive with state-of-the-art methods SAE (Tu et al, 2020) and HGN (Fang et al, 2019) that use additional, strong supervision on which sentences are relevant to the question. Second, our analysis shows that sub-questions improve multi-hop QA by using the single-hop QA model to retrieve question-relevant text.…”

Section: Seq2seq Ormentioning

confidence: 89%

Unsupervised Question Decomposition for Question Answering

Perez¹,

Lewis²,

Yih³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

We aim to improve question answering (QA) by decomposing hard questions into simpler sub-questions that existing QA systems are capable of answering. Since labeling questions with decompositions is cumbersome, we take an unsupervised approach to produce sub-questions, also enabling us to leverage millions of questions from the internet. Specifically, we propose an algorithm for One-to-N Unsupervised Sequence transduction (ONUS) that learns to map one hard, multi-hop question to many simpler, singlehop sub-questions. We answer sub-questions with an off-the-shelf QA model and give the resulting answers to a recomposition model that combines them into a final answer. We show large QA improvements on HOTPOTQA over a strong baseline on the original, out-ofdomain, and multi-hop dev sets. ONUS automatically learns to decompose different kinds of questions, while matching the utility of supervised and heuristic decomposition methods for QA and exceeding those methods in fluency. Qualitatively, we find that using subquestions is promising for shedding light on why a QA system makes a prediction. 1 * KC was a part-time research scientist at Facebook AI Research while working on this paper.1 Our code, data, and pretrained models are available at https://github.com/facebookresearch/ UnsupervisedDecomposition. What profession do H. L. Mencken and Albert Camus have in common? Unsupervised Decomp. Model (ONUS) What profession does H. L. Mencken have? Single Hop QA Model Single Hop QA Model Henry Louis Mencken (1880 -1956) was an American journalist, critic and scholar of American English.

show abstract

“…Datasets D and T(D): HotpotQA is a popular multi-hop QA dataset with about 113K questions which has spurred many models (Nishida et al, 2019;Xiao et al, 2019;Tu et al, 2020;Fang et al, 2020). We use the distractor setting where each question has a set of 10 input paragraphs, of which two were used to create the multifact question.…”

Section: Methodsmentioning

confidence: 99%

“…Multi-hop Reasoning: Many multifact reasoning approaches have been proposed for HotpotQA and similar datasets (Mihaylov et al, 2018;Khot et al, 2020). These use iterative fact selection (Nishida et al, 2019;Tu et al, 2020;Asai et al, 2020;Das et al, 2019), graph neural networks (Xiao et al, 2019;Fang et al, 2020;Tu et al, 2020), or simply cross-document self-attention (Yang et al, 2019;Beltagy et al, 2020) to capture inter-paragraph interaction. While these approaches have pushed the state of the art, the extent of actual progress on multifact reasoning remains unclear.…”

Section: Reducing Disconnected Reasoningmentioning

confidence: 99%

Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning

Trivedi

Balasubramanian

Khot³

et al. 2020

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)

View full text Add to dashboard Cite

Has there been real progress in multi-hop question-answering? Models often exploit dataset artifacts to produce correct answers, without connecting information across multiple supporting facts. This limits our ability to measure true progress and defeats the purpose of building multi-hop QA datasets. We make three contributions towards addressing this. First, we formalize such undesirable behavior as disconnected reasoning across subsets of supporting facts. This allows developing a model-agnostic probe for measuring how much any model can cheat via disconnected reasoning. Second, using a notion of contrastive support sufficiency, we introduce an automatic transformation of existing datasets that reduces the amount of disconnected reasoning. Third, our experiments 1 suggest that there hasn't been much progress in multifact QA in the reading comprehension setting. For a recent large-scale model (XLNet), we show that only 18 points out of its answer F1 score of 72 on HotpotQA are obtained through multifact reasoning, roughly the same as that of a simpler RNN baseline. Our transformation substantially reduces disconnected reasoning (19 points in answer F1). It is complementary to adversarial approaches, yielding further reductions in conjunction. * Early portion of this work was done during the first author's internship at Allen Institute for AI. 1 https://github.com/stonybrooknlp/dire Which country got independence when the cold war started? Supporting Facts (SF)

show abstract

Hierarchical Graph Network for Multi-hop Question Answering

Cited by 100 publications

References 39 publications

A Simple Yet Strong Pipeline for HotpotQA

A Simple Yet Strong Pipeline for HotpotQA

Unsupervised Question Decomposition for Question Answering

Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning

Contact Info

Product

Resources

About