Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation

Abstract

The Transformer architecture has led to significant gains in machine translation. However, most studies focus on only sentence-level translation without considering the context dependency within documents, leading to the inadequacy of document-level coherence. Some recent research tried to mitigate this issue by introducing an additional context encoder or translating with multiple sentences or even the entire document. Such methods may lose the information on the target side or have an increasing computational complexity as documents get longer. To address such problems, we introduce a recurrent memory unit to the vanilla Transformer, which supports the information exchange between the sentence and previous context. The memory unit is recurrently updated by acquiring information from sentences, and passing the aggregated knowledge back to subsequent sentence states. We follow a two-stage training strategy, in which the model is first trained at the sentence level and then finetuned for document-level translation. We conduct experiments on three popular datasets for document-level machine translation and our model has an average improvement of 0.91 s-BLEU over the sentence-level baseline. We also achieve state-of-the-art results on TED and News, outperforming the previous work by 0.36 s-BLEU and 1.49 d-BLEU on average.

Document-Level Neural Machine Translation With Recurrent Context States

“…In this paper, we focus on the speed-up of context-aware NMT when the global context is involved. Note that Wang et al [37] and Feng et al [51] both use recurrent networks to capture contextual information, which is similar to our work. Compared with Wang et al [37]'s work, we treat the document translation as a continuous process.…”

“…The memory unit stored contextual information with multiple vectors and it was updated by the extra attention modules of each layer in both the encoder and decoder. Compared with Feng et al [51]'s work, our work is more efficient because of the simplicity of the architecture. In our work, no additional modules are required other than the RNN units.…”

“…Some researchers also considered introducing graph neural networks into context-aware NMT [48], [49]. More recently, researchers took a step towards extracting the global contextual information from the whole document [16], [50], [51].…”

“…So we do not need to re-compute the contextual representation when translating a new sentence in the same document. Feng et al [51] introduced a contextual memory unit and two attention modules to manipulate the memory. The memory unit stored contextual information with multiple vectors and it was updated by the extra attention modules of each layer in both the encoder and decoder.…”