Learn To Remember: Transformer with Recurrent Memory for Document-Level Machine Translation

Abstract

The Transformer architecture has led to significant gains in machine translation. However, most studies focus on only sentence-level translation without considering the context dependency within documents, leading to the inadequacy of document-level coherence. Some recent research tried to mitigate this issue by introducing an additional context encoder or translating with multiple sentences or even the entire document. Such methods may lose the information on the target side or have an increasing computational complexity as documents get longer. To address such problems, we introduce a recurrent memory unit to the vanilla Transformer, which supports the information exchange between the sentence and previous context. The memory unit is recurrently updated by acquiring information from sentences, and passing the aggregated knowledge back to subsequent sentence states. We follow a two-stage training strategy, in which the model is first trained at the sentence level and then finetuned for document-level translation. We conduct experiments on three popular datasets for document-level machine translation and our model has an average improvement of 0.91 s-BLEU over the sentence-level baseline. We also achieve state-of-the-art results on TED and News, outperforming the previous work by 0.36 s-BLEU and 1.49 d-BLEU on average.

Exploring Discourse Structure in Document-level Machine Translation

“…Following previous work (Bao et al, 2021;Sun et al, 2022;Feng et al, 2022), we apply sentence-level BLEU score (s-BLEU) and document-level BLEU score (d-BLEU) as the metrics of evaluation. Since our methods are focused on the DocMT and do not involve sentence alignments, the d-BLEU score is our major metric, which matches n-grams in the whole document.…”

“…Among these methods, the dominant approaches still adhere to the sentence-by-sentence mode, but they utilize additional contextual information, including the surrounding sentences Miculicich et al, 2018;Kang et al, 2020;Zhang et al, 2020bZhang et al, , 2021a, document contextual representation (Jiang et al, 2020;Ma et al, 2020) and memory units (Feng et al, 2022). In recent years, many researches have turned to translating multiple sentences or the entire document at once (Tan et al, 2019;Bao et al, 2021;Sun et al, 2022;.…”

CUNI-KIT System for Simultaneous Speech Translation Task at IWSLT 2022

“…Dai et al (2019) introduced a recurrence mechanism and improved positional encoding scheme in the Transformer. Later work proposed an explicit compressed memory realized by a few dense vectors (Feng et al, 2022).…”

“…As pointed out in Sections 3.2 to 3.4, the performance usually drops with a context longer than a few sentences. Some solutions have been suggested (Kim et al, 2019;Feng et al, 2022), but it remains unclear how to adapt these approaches for SST with the specifics of SST in mind (e.g., computational constraints, speech input).…”

A Detection-based Attention Alignment Method for Document-level Neural Machine Translation

“…During the last decade, neural machine translation (NMT) has made remarkable progress to become a stateof-the-art method, especially for sentence-level translation [1,2]. In document-level translation, it is widely accepted that the introduction of discourse dependencies between sentences can improve the coherence and quality of the DOI reference number: 10.18293/SEKE2023-165 translated text [3,4]. Like those for sentence-level NMT, most existing document-level NMT (DocNMT) models integrate contextual information using an attention mechanism.…”