Predicting Target Language CCG Supertags Improves Neural Machine Translation

Nădejde, Maria; Reddy, Siva; Sennrich, Rico; Dwojak, Tomasz; Junczys-Dowmunt, Marcin; Koehn, Philipp; Birch, Alexandra

doi:10.18653/v1/w17-4707

Cited by 62 publications

(51 citation statements)

References 26 publications

(37 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In contrast to these approaches, the DSA-LSTM only models the probability of surface strings, albeit with an auxiliary loss that distills the next-word predictive distribution of a syntactic language model. Earlier work has also explored multi-task learning with syntactic objectives as an auxiliary loss in language modelling and machine translation (Luong et al, 2016;Eriguchi et al, 2016;Nadejde et al, 2017;Enguehard et al, 2017;Aharoni and Goldberg, 2017;Eriguchi et al, 2017). Our approach of injecting syntactic bias through a KD objective is orthogonal to this approach, with the primary difference that here the student DSA-LSTM has no direct access to syntactic annotations; it does, however, have access to the teacher RNNG's softmax distribution over the next word.…”

Section: Related Workmentioning

confidence: 99%

Scalable Syntax-Aware Language Models Using Knowledge Distillation

Kuncoro

Dyer

Rimell

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

Prior work has shown that, on small amounts of training data, syntactic neural language models learn structurally sensitive generalisations more successfully than sequential language models. However, their computational complexity renders scaling difficult, and it remains an open question whether structural biases are still necessary when sequential models have access to ever larger amounts of training data. To answer this question, we introduce an efficient knowledge distillation (KD) technique that transfers knowledge from a syntactic language model trained on a small corpus to an LSTM language model, hence enabling the LSTM to develop a more structurally sensitive representation of the larger training data it learns from. On targeted syntactic evaluations, we find that, while sequential LSTMs perform much better than previously reported, our proposed technique substantially improves on this baseline, yielding a new state of the art. Our findings and analysis affirm the importance of structural biases, even in models that learn from large amounts of data.

show abstract

Section: Related Workmentioning

confidence: 99%

Scalable Syntax-Aware Language Models Using Knowledge Distillation

Kuncoro

Dyer

Rimell

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…Section 10 sheds light on the overall patterns that arise from Table 1: Example sentence with different word-level annotations. The CCG supertags are taken from Nadejde et al (2017). POS and semantic tags are our own annotation, as well as the German translation and its morphological tags.…”

Section: Introductionmentioning

confidence: 99%

On the Linguistic Representational Power of Neural Machine Translation Models

Belinkov

Durrani²,

Dalvi³

et al. 2020

Computational Linguistics

View full text Add to dashboard Cite

Despite the recent success of deep neural networks in natural language processing (NLP) and other spheres of artificial intelligence (AI), their interpretability remains a challenge. We analyze the representations learned by neural machine translation (NMT) models at various levels of granularity and evaluate their quality through relevant extrinsic properties. In particular, we seek answers to the following questions: (i) How accurately is word-structure captured within the learned representations, which is an important aspect in translating morphologically-rich languages? (ii) Do the representations capture long-range dependencies, and effectively handle syntactically divergent languages? (iii) Do the representations capture lexical semantics? We conduct a thorough investigation along several parameters: (i) Which layers in the architecture capture each of these linguistic phenomena; (ii) How does the choice of translation unit (word, character, or subword unit) impact the linguistic properties captured by the underlying representations? (iii) Do the encoder and decoder learn differently and independently? (iv) Do the representations learned by multilingual NMT models capture the same amount of linguistic information as their bilingual counterparts? Our data-driven, quantitative evaluation illuminates important aspects in NMT models and their ability to capture various linguistic phenomena. We show that deep NMT models trained in an end-to-end fashion, without being provided any direct supervision during the training process, learn a non-trivial amount of linguistic information. Notable findings include the following observations: i) Word morphology and part-of-speech information are captured at the lower layers of the model; (ii) In contrast, lexical semantics or non-local syntactic and semantic dependencies are better represented at the higher layers of the model; (iii) Representations learned using characters are more informed about wordmorphology compared to those learned using subword units; and (iv) Representations learned by multilingual models are richer compared to bilingual models.

show abstract

“…NMT systems have outperformed the state-of-the-art SMT model on various language pairs in terms of translation qual-ity (Luong et al, 2015;Bentivogli et al, 2016;Junczys-Dowmunt et al, 2016;Wu et al, 2016;Toral and Sánchez-Cartagena, 2017). However, due to some deficiencies of NMT systems such as the limited vocabulary size, low adequacy for some translations, much research work has involved incorporating extra knowledge such as SMT features or linguistic features into NMT to improve translation performance (He et al, 2016;Sennrich and Haddow, 2016;Nadejde et al, 2017;Wang et al, 2017).…”

Section: Introductionmentioning

confidence: 99%

Pre-Reordering for Neural Machine Translation: Helpful or Harmful?

Way

2017

The Prague Bulletin of Mathematical Linguistics

View full text Add to dashboard Cite

Pre-reordering, a preprocessing to make the source-side word orders close to those of the target side, has been proven very helpful for statistical machine translation (SMT) in improving translation quality. However, is it the case in neural machine translation (NMT)? In this paper, we firstly investigate the impact of pre-reordered source-side data on NMT, and then propose to incorporate features for the pre-reordering model in SMT as input factors into NMT (factored NMT). The features, namely parts-of-speech (POS), word class and reordered index, are encoded as feature vectors and concatenated to the word embeddings to provide extra knowledge for NMT. Pre-reordering experiments conducted on Japanese↔English and Chinese↔English show that pre-reordering the source-side data for NMT is redundant and NMT models trained on pre-reordered data deteriorate translation performance. However, factored NMT using SMT-based pre-reordering features on Japanese→English and Chinese→English is beneficial and can further improve by 4.48 and 5.89 relative BLEU points, respectively, compared to the baseline NMT system.

show abstract

Predicting Target Language CCG Supertags Improves Neural Machine Translation

Cited by 62 publications

References 26 publications

Scalable Syntax-Aware Language Models Using Knowledge Distillation

Scalable Syntax-Aware Language Models Using Knowledge Distillation

On the Linguistic Representational Power of Neural Machine Translation Models

Pre-Reordering for Neural Machine Translation: Helpful or Harmful?

Contact Info

Product

Resources

About