Multiscale Collaborative Deep Models for Neural Machine Translation

Abstract

Recent evidence reveals that Neural Machine Translation (NMT) models with deeper neural networks can be more effective but are difficult to train. In this paper, we present a MultiScale Collaborative (MSC) framework to ease the training of NMT models that are substantially deeper than those used previously. We explicitly boost the gradient backpropagation from top to bottom levels by introducing a block-scale collaboration mechanism into deep NMT models. Then, instead of forcing the whole encoder stack directly learns a desired representation, we let each encoder block learns a fine-grained representation and enhance it by encoding spatial dependencies using a context-scale collaboration. We provide empirical evidence showing that the MSC nets are easy to optimize and can obtain improvements of translation quality from considerably increased depth. On IWSLT translation tasks with three translation directions, our extremely deep models (with 72-layer encoders) surpass strong baselines by +2.2∼+3.1 BLEU points. In addition, our deep MSC achieves a BLEU score of 30.56 on WMT14 English→German task that significantly outperforms state-of-the-art deep NMT models.

ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation

“…Wide models can also benefit from the enlarging layer depth (Wei et al, 2020;. The RK-2 ODE Transformer achieves BLEU score of 30.76 and 44.11 on the En-De and the En-Fr tasks, significantly surpassing the standard Big model by 1.32 and 0.70 BLEU points.…”

“…Deep Transformer models: Recently, deep Transformer has witnessed tremendous success in machine translation. A straightforward way is to shorten the path from upper-level layers to lowerlevel layers thus to alleviate the gradient vanishing or exploding problems (Bapna et al, 2018;Wu et al, 2019;Wei et al, 2020). For deeper models, the training cost is nonnegligible.…”

“…These problems make the situation worse, when more layers are stacked and errors are propagated through the neural network. It might explain why recent Machine Translation (MT) systems cannot benefit from extremely deep models Wei et al, 2020;.…”

Domain Adaptation and Multi-Domain Adaptation for Neural Machine Translation: A Survey

“…In terms of domain adaptation, as models become larger (Wei et al, 2020b) and as high quality personalization becomes more important to users (Buj et al, 2020), there may be a growing focus on approaches which do not adapt the entire model. Instead, we predict that memory efficiency for tuning and storing models will continue to grow in importance.…”

Shallow-to-Deep Training for Neural Machine Translation

“…More stacked layers lead to a stronger model of representing the sentence. This particularly makes sense in the deep NMT scenario because it has been proven that deep models can benefit from an enriched representation Wu et al, 2019b;Wei et al, 2020).…”