Contextualized Non-local Neural Networks for Sequence Learning

Liu, Pengfei; Chang, Shuaichen; Huang, Xuanjing; Tang, Jian; Cheung, Jackie Chi Kit

doi:10.48550/arxiv.1811.08600

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(Norcliffe-Brown et al, 2018) dynamically construct a graph which contains all the visual objects appearing in an image. In parallel to our work, (Liu et al, 2018) also dynamically construct a graph of all words from free text.…”

Section: Graph Neural Networkmentioning

confidence: 99%

GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension

Chen

Wu²,

Zaki

2020

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

Conversational machine comprehension (MC) has proven significantly more challenging compared to traditional MC since it requires better utilization of conversation history. However, most existing approaches do not effectively capture conversation history and thus have trouble handling questions involving coreference or ellipsis. Moreover, when reasoning over passage text, most of them simply treat it as a word sequence without exploring rich semantic relationships among words. In this paper, we first propose a simple yet effective graph structure learning technique to dynamically construct a question and conversation history aware context graph at each conversation turn. Then we propose a novel Recurrent Graph Neural Network, and based on that, we introduce a flow mechanism to model the temporal dependencies in a sequence of context graphs. The proposed GraphFlow model can effectively capture conversational flow in a dialog, and shows competitive performance compared to existing state-of-the-art methods on CoQA, QuAC and DoQA benchmarks. In addition, visualization experiments show that our proposed model can offer good interpretability for the reasoning process.

show abstract

Section: Graph Neural Networkmentioning

confidence: 99%

GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension

Chen

Wu²,

Zaki

2020

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence

View full text Add to dashboard Cite

show abstract

“…The success of Transformer has raised a large body of follow-up work. Therefore, some Transformer variations are also proposed, such as GPT (Radford et al, 2018), BERT (Devlin et al, 2018), Transformer-XL (Dai et al, 2019) , Universal Transformer (Dehghani et al, 2018) and CN 3 (Liu et al, 2018a).…”

Section: Modelling Non-local Compositionalitymentioning

confidence: 99%

Star-Transformer

Guo

Qiu

Liu

et al. 2019

Proceedings of the 2019 Conference of the North

Self Cite

154

View full text Add to dashboard Cite

Although Transformer has achieved great successes on many NLP tasks, its heavy structure with fully-connected attention connections leads to dependencies on large training data. In this paper, we present Star-Transformer, a lightweight alternative by careful sparsification. To reduce model complexity, we replace the fully-connected structure with a star-shaped topology, in which every two non-adjacent nodes are connected through a shared relay node. Thus, complexity is reduced from quadratic to linear, while preserving capacity to capture both local composition and long-range dependency. The experiments on four tasks (22 datasets) show that Star-Transformer achieved significant improvements against the standard Transformer for the modestly sized datasets.

show abstract

“…GNN for NLP: Recently, there is considerable amount of interest in applying GNN to NLP tasks and great success has been achieved. For example, in neural machine translation, GNN has been employed to integrate syntactic and semantic information into encoders (Bastings et al, 2017;Marcheggiani et al, 2018); applied GNN to relation extraction over pruned dependency trees; the study by Yao et al (2018) employed GNN over a heterogeneous graph to do text classification, which inspires our idea of the HDE graph; Liu et al (2018) proposed a new contextualized neural network for sequence learning by leveraging various types of non-local contextual information in the form of information passing over GNN. These studies are related to our work in the sense that we both use GNN to improve the information interaction over long context or across documents.…”

Section: Multi-hop Rcmentioning

confidence: 99%

Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs

Tao¹,

Wang²,

Huang³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

135

View full text Add to dashboard Cite

Multi-hop reading comprehension (RC) across documents poses new challenge over singledocument RC because it requires reasoning over multiple documents to reach the final answer. In this paper, we propose a new model to tackle the multi-hop RC problem. We introduce a heterogeneous graph with different types of nodes and edges, which is named as Heterogeneous Document-Entity (HDE) graph. The advantage of HDE graph is that it contains different granularity levels of information including candidates, documents and entities in specific document contexts. Our proposed model can do reasoning over the HDE graph with nodes representation initialized with co-attention and self-attention based context encoders. We employ Graph Neural Networks (GNN) based message passing algorithms to accumulate evidences on the proposed HDE graph. Evaluated on the blind test set of the Qangaroo WIKIHOP data set, our HDE graph based single model delivers competitive result, and the ensemble model achieves the state-of-the-art performance.

show abstract

Contextualized Non-local Neural Networks for Sequence Learning

Cited by 4 publications

References 0 publications

GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension

GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension

Star-Transformer

Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs

Contact Info

Product

Resources

About