Graph Convolutional Encoders for Syntax-aware Neural Machine Translation

Abstract

We present a simple and effective approach to incorporating syntactic structure into neural attention-based encoderdecoder models for machine translation. We rely on graph-convolutional networks (GCNs), a recent class of neural networks developed for modeling graph-structured data. Our GCNs use predicted syntactic dependency trees of source sentences to produce representations of words (i.e. hidden states of the encoder) that are sensitive to their syntactic neighborhoods. GCNs take word representations as input and produce word representations as output, so they can easily be incorporated as layers into standard encoders (e.g., on top of bidirectional RNNs or convolutional neural networks). We evaluate their effectiveness with English-German and English-Czech translation experiments for different types of encoders and observe substantial improvements over their syntax-agnostic versions in all the considered setups.

Graph-based Neural Sentence Ordering

“…Graph Neural Networks in NLP. Recently, graph neural networks have been shown successful in the NLP community, such as modeling semantic graphs [Beck et al, 2018;Song et al, 2018a;Song et al, 2019], dependency trees Bastings et al, 2017;Song et al, 2018b], knowledge graphs and even sentences Xu et al, 2018]. Particularly, Zhang et al, [2018] proposed GRN to represent raw sentences by building a graph structure of neighboring words and a sentence-level node.…”

Dual-Attention Graph Convolutional Network

“…There are some research coming to explore the graph convolutional work that are more suitable for text classification. Firstly GCNs are used to capture the syntactic structure in [3], which produce representations of words and show the improvement. The method [13] mentioned in the last paragraph apply GCN to text classification, but it can't naturally support edge features.…”

“…However, these deep neural networks cannot well model the irregular structure of texts, which is crucial for text recognition task. Recently, graph convolutional networks (GCNs) [3,13] have been proposed with a lot of success in various tasks, and also applied in feature representation of texts. On the other hand, due to the difficulty in modeling data variance, the attention mechanism [18,2,23] is proposed and widely embedded in multiple models, achieving promising results on a variety of tasks.…”

Higher-Order Syntactic Attention Network for Longer Sentence Compression

“…Similar to HiSAN, Hashimoto and Tsuruoka (2017) use dependency features as attention distributions, but different from HiSAN, they use pre-trained dependency relations, and do not take into account the chains of dependencies. ; Bastings et al (2017) consider higherorder dependency relationships in Seq2Seq by incorporating a graph convolution technique (Kipf and Welling, 2016) into the encoder. However, the dependency information of the graph convolution technique is still given in pipeline manner.…”