Montreal Neural Machine Translation Systems for WMT’15

Jean, Sébastien; Fırat, Orhan; Cho, Kyunghyun; Memisevic, Roland; Bengio, Yoshua

doi:10.18653/v1/w15-3014

Cited by 120 publications

(102 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In deep fusion, the controller parameters and output parameters are tuned on further parallel training data, but the language model parameters are fixed during the finetuning stage. Jean et al (2015b) also report on experiments with reranking of NMT output with a 5-gram language model, but improvements are small (between 0.1-0.5 BLEU).…”

Section: Related Workmentioning

confidence: 99%

Improving Neural Machine Translation Models with Monolingual Data

Sennrich¹,

Haddow²,

Birch³

2016

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

1,881

1,811

View full text Add to dashboard Cite

Neural Machine Translation (NMT) has obtained state-of-the art performance for several language pairs, while only using parallel data for training. Targetside monolingual data plays an important role in boosting fluency for phrasebased statistical machine translation, and we investigate the use of monolingual data for NMT. In contrast to previous work, which combines NMT models with separately trained language models, we note that encoder-decoder NMT architectures already have the capacity to learn the same information as a language model, and we explore strategies to train with monolingual data without changing the neural network architecture. By pairing monolingual training data with an automatic backtranslation, we can treat it as additional parallel training data, and we obtain substantial improvements on the WMT 15 task English↔German (+2.8-3.7 BLEU), and for the low-resourced IWSLT 14 task Turkish→English (+2.1-3.4 BLEU), obtaining new state-of-the-art results. We also show that fine-tuning on in-domain monolingual and parallel data gives substantial improvements for the IWSLT 15 task English→German.

show abstract

Section: Related Workmentioning

confidence: 99%

Improving Neural Machine Translation Models with Monolingual Data

Sennrich¹,

Haddow²,

Birch³

2016

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

1,881

1,811

View full text Add to dashboard Cite

show abstract

“…While this monotonicity condition is true for vanilla NMT (Eq. 3), it does not hold for methods like length normalization (Jean et al, 2015;Boulanger-Lewandowski et al, 2013; or word rewards (He et al, 2016): Length normalization gives an advantage to longer hypotheses by dividing the score by the sentence length, while a word reward directly violates monotonicity as it rewards each word with a positive value. In Sec.…”

Section: Exact Inference For Neural Modelsmentioning

confidence: 99%

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

Stahlberg¹,

Byrne²

2019

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen

100

128

View full text Add to dashboard Cite

We report on search errors and model errors in neural machine translation (NMT). We present an exact inference procedure for neural sequence models based on a combination of beam search and depth-first search. We use our exact search to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set. Surprisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100. For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy. We show by constraining search with a minimum translation length that at the root of the problem of empty translations lies an inherent bias towards shorter translations. We conclude that vanilla NMT in its current form requires just the right amount of beam search errors, which, from a modelling perspective, is a highly unsatisfactory conclusion indeed, as the model often prefers an empty translation.

show abstract

“…We call this averaging averagedbest. On the other hand, building ensembles requires training several models which is time consuming, however, it is common to do that in NMT (Jean et al, 2015). Thus, an investigation is needed to discover whether either the model or the estimation of its parameters is weak.…”

Section: Introductionmentioning

confidence: 99%

Empirical Investigation of Optimization Algorithms in Neural Machine Translation

Bahar

Alkhouli

Peter

et al. 2017

The Prague Bulletin of Mathematical Linguistics

View full text Add to dashboard Cite

Training neural networks is a non-convex and a high-dimensional optimization problem. In this paper, we provide a comparative study of the most popular stochastic optimization techniques used to train neural networks. We evaluate the methods in terms of convergence speed, translation quality, and training stability. In addition, we investigate combinations that seek to improve optimization in terms of these aspects. We train state-of-the-art attention-based models and apply them to perform neural machine translation. We demonstrate our results on two tasks: WMT 2016 En→Ro and WMT 2015 De→En.

show abstract

Montreal Neural Machine Translation Systems for WMT’15

Cited by 120 publications

References 11 publications

Improving Neural Machine Translation Models with Monolingual Data

Improving Neural Machine Translation Models with Monolingual Data

On NMT Search Errors and Model Errors: Cat Got Your Tongue?

Empirical Investigation of Optimization Algorithms in Neural Machine Translation

Contact Info

Product

Resources

About