Question Directed Graph Attention Network for Numerical Reasoning over Text

Chen, Kunlong; Xu, Weidi; Cheng, Xingyi; Xiaochuan, Zou; Zhang, Yuyu; Song, Le; Wang, Taifeng; Yuan, Qi; Chu, Wei

doi:10.18653/v1/2020.emnlp-main.549

Cited by 39 publications

(52 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…HybridQA (Chen et al, 2020b) is one existing hybrid dataset for QA tasks, where the context is a table connected with Wiki pages via hyperlinks. Numerical Reasoning Numerical reasoning is key to many NLP tasks like question answering (Dua et al, 2019;Ran et al, 2019;Andor et al, 2019;Chen et al, 2020a;Pasupat and Liang, 2015;Herzig et al, 2020;Yin et al, 2020; and arithmetic word problems (Kushman et al, 2014;Mitra and Baral, 2016;Huang et al, 2017;Ling et al, 2017). To our best knowledge, no prior work attempts to develop models able to perform numerical reasoning over hybrid contexts.…”

Section: Results and Analysismentioning

confidence: 99%

TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

Zhu¹,

Lei²,

Huang³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

Hybrid data combining both tabular and textual content (e.g., financial reports) are quite pervasive in the real world. However, Question Answering (QA) over such hybrid data is largely neglected in existing research. In this work, we extract samples from real financial reports to build a new large-scale QA dataset containing both Tabular And Textual data, named TAT-QA, where numerical reasoning is usually required to infer the answer, such as addition, subtraction, multiplication, division, counting, comparison/sorting, and their compositions. We further propose a novel QA model termed TAGOP, which is capable of reasoning over both tables and text. It adopts sequence tagging to extract relevant cells from the table along with relevant spans from the text to infer their semantics, and then applies symbolic reasoning over them with a set of aggregation operators to arrive at the final answer. TAGOP achieves 58.0% in F 1 , which is an 11.1% absolute increase over the previous best baseline model, according to our experiments on TAT-QA. But this result still lags far behind the performance of human expert, i.e. 90.8% in F 1 . It demonstrates that our TAT-QA is very challenging and can serve as a benchmark for training and testing powerful QA models that address hybrid data. Our dataset is publicly available for noncommercial use at https://nextplusplus. github.io/TAT-QA/.

show abstract

Section: Results and Analysismentioning

confidence: 99%

TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

Zhu¹,

Lei²,

Huang³

et al. 2021

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer

View full text Add to dashboard Cite

show abstract

“…GenBERT (Geva et al, 2020) pre-trains BERT (Devlin et al, 2019) with synthetic number and text data. QDGAT (Chen et al, 2020) designs a graph neural network with fully-connected number nodes of same entity type. While there are many other related works on this topic (Hu et al, 2019;Andor et al, 2019;Gupta et al, 2019;Min et al, 2019;Sundararaman et al, 2020;Saha et al, 2021), none of them address the problem of extrapolation in DROP.…”

Section: Related Workmentioning

confidence: 99%

“…Models To inspect the extrapolation capability among the existing models in DROP, we evaluate the following representative models in the leaderboard: NAQANet (the official baseline model in DROP), NumNet (Ran et al, 2019), Num-Net+(RoBERTa) and GenBERT (Geva et al, 2020). Although we mention QDGAT (Chen et al, 2020) in this paper, we did not evaluate it because its official implementation could not be reproduced.…”

Section: Empirical Investigation On Extrapolationmentioning

confidence: 99%

“…The research in question-answering (QA) models that are able to perform reading comprehension and discrete reasoning over numbers in the passage has seen significant progress, like the models in DROP (Ran et al, 2019;Hu et al, 2019;Chen et al, 2020;Geva et al, 2020). Despite their ability to understand the complex context and numbers within, none of them deal with the notion of whether these models can robustly handle "unseen" numbers during testing.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Have You Seen That Number? Investigating Extrapolation in Question Answering Models

Kim¹,

Hong²,

Kyung-min³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Numerical reasoning in machine reading comprehension (MRC) has shown drastic improvements over the past few years. While the previous models for numerical MRC are able to interpolate the learned numerical reasoning capabilities, it is not clear whether they can perform just as well on numbers unseen in the training dataset. Our work rigorously tests state-of-the-art models on DROP, a numerical MRC dataset, to see if they can handle passages that contain out-of-range numbers. One of the key findings is that the models fail to extrapolate to unseen numbers. Presenting numbers as digit-by-digit input to the model, we also propose the E-digit number form that alleviates the lack of extrapolation in models and reveals the need to treat numbers differently from regular words in the text. Our work provides a valuable insight into the numerical MRC models and the way to represent number forms in MRC. * Equal contribution.As of the census of 2000, there were 49,927 people, 18,009 households, and 12,192 families residing in the county. The population density was 48 people per square mile (19/km²).As of the census of 2000, there were 4,992,700 people, 1,800,900 households, and 1,219,200 families residing in the county. The population density was 4,800 people per square mile (19/km²).

show abstract

“…NumNet(Ran et al, 2019) leveraged Graph Neural Network (GNN) to design a number-aware deep learning model. Also leveraging GNN,Chen et al (2020a) distinguished number types more precisely by adding the connection with entities and obtained better performance Chen et al (2020b). searched possible programs exhaustively based on answer numbers and employed these programs as weak supervision to train the whole model.…”

mentioning

confidence: 99%

Improving Numerical Reasoning Skills in the Modular Approach for Complex Question Answering on Text

Guo¹,

Haffari

2021

Findings of the Association for Computational Linguistics: EMNLP 2021

View full text Add to dashboard Cite

Numerical reasoning skills are essential for complex question answering (CQA) over text. It requires opertaions including counting, comparison, addition and subtraction. A successful approach to CQA on text, Neural Module Networks (NMNs), follows the programmer-interpreter paradigm and leverages specialised modules to perform compositional reasoning. However, the NMNs framework does not consider the relationship between numbers and entities in both questions and paragraphs. We propose effective techniques to improve NMNs' numerical reasoning capabilities by making the interpreter questionaware and capturing the relationship between entities and numbers. On the same subset of the DROP dataset for CQA on text, experimental results show that our additions outperform the original NMNs by 3.0 points for the overall F1 score.

show abstract

Question Directed Graph Attention Network for Numerical Reasoning over Text

Cited by 39 publications

References 17 publications

TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

Have You Seen That Number? Investigating Extrapolation in Question Answering Models

Improving Numerical Reasoning Skills in the Modular Approach for Complex Question Answering on Text

Contact Info

Product

Resources

About