A Survey on Document-level Neural Machine Translation: Methods and Evaluation

Maruf, Sameen; Saleh, Fahimeh; Haffari, Gholamreza

doi:10.48550/arxiv.1912.08494

Cited by 9 publications

(9 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, despite being sparse, these few cases strongly impact the quality of translation (Läubli et al, 2018;Popescu-Belis, 2019). As a consequence, a number of discourse-targeted test sets and automatic metrics have been proposed to measure improvements in context-aware MT (Maruf et al, 2019b), the most widely adopted ones being contrastive test sets.…”

Section: Targeted Evaluationmentioning

confidence: 99%

Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models

Lupo¹,

Dinarelli²,

Besacier³

2021

Preprint

View full text Add to dashboard Cite

Multi-encoder models are a broad family of context-aware Neural Machine Translation (NMT) systems that aim to improve translation quality by encoding document-level contextual information alongside the current sentence. The context encoding is undertaken by contextual parameters, trained on documentlevel data. In this work, we show that training these parameters takes large amount of data, since the contextual training signal is sparse. We propose an efficient alternative, based on splitting sentence pairs, that allows to enrich the training signal of a set of parallel sentences by breaking intra-sentential syntactic links, and thus frequently pushing the model to search the context for disambiguating clues. We evaluate our approach with BLEU and contrastive test sets, showing that it allows multi-encoder models to achieve comparable performances to a setting where they are trained with ×10 document-level data. We also show that our approach is a viable option to context-aware NMT for language pairs with zero document-level parallel data.

show abstract

Section: Targeted Evaluationmentioning

confidence: 99%

Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models

Lupo¹,

Dinarelli²,

Besacier³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In spite of its success (Vaswani et al, 2017), sentence-level NMT has been based on strong independence and locality assumptions generally, in which the interrelations among these discourse (Jurafsky, 2000) elements were ignored. This results in that the translations may be perfect at the sentence-level but lack crucial properties of the text, hindering understanding (Maruf et al, 2019).…”

Section: Document-enhanced Nmtmentioning

confidence: 99%

SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task

Zhao

Wang

et al. 2020

Preprint

View full text Add to dashboard Cite

In this paper, we introduced our joint team SJTU-NICT 's participation in the WMT 2020 machine translation shared task. In this shared task, we participated in four translation directions of three language pairs: English-Chinese, English-Polish on supervised machine translation track, German-Upper Sorbian on lowresource and unsupervised machine translation tracks. Based on different conditions of language pairs, we have experimented with diverse neural machine translation (NMT) techniques: document-enhanced NMT, XLM pre-trained language model enhanced NMT, bidirectional translation as a pre-training, reference language based UNMT, data-dependent gaussian prior objective, and BT-BLEU collaborative filtering self-training. We also used the TF-IDF algorithm to filter the training set to obtain a domain more similar set with the test set for finetuning. In our submissions, the primary systems won the first place on English to Chinese, Polish to English, and German to Upper Sorbian translation directions.

show abstract

“…From a practical computational efficiency standpoint, this is unsatisfactory. Indeed, consider the problem of document-level machine translation [10]. When solving this problem using a vanilla Transformer, one has to choose N large enough to accurately translate the longest document in the training set.…”

Section: Relevant Aspects Of the Vanilla Transformermentioning

confidence: 99%

N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equations

Baier-Reinio,

De Sterck

2020

Preprint

View full text Add to dashboard Cite

We use neural ordinary differential equations to formulate a variant of the Transformer that is depth-adaptive in the sense that an input-dependent number of time steps is taken by the ordinary differential equation solver. Our goal in proposing the N-ODE Transformer is to investigate whether its depth-adaptivity may aid in overcoming some specific known theoretical limitations of the Transformer in handling nonlocal effects. Specifically, we consider the simple problem of determining the parity of a binary sequence, for which the standard Transformer has known limitations that can only be overcome by using a sufficiently large number of layers or attention heads. We find, however, that the depth-adaptivity of the N-ODE Transformer does not provide a remedy for the inherently nonlocal nature of the parity problem, and provide explanations for why this is so. Next, we pursue regularization of the N-ODE Transformer by penalizing the arclength of the ODE trajectories, but find that this fails to improve the accuracy or efficiency of the N-ODE Transformer on the challenging parity problem. We suggest future avenues of research for modifications and extensions of the N-ODE Transformer that may lead to improved accuracy and efficiency for sequence modelling tasks such as neural machine translation.

show abstract

A Survey on Document-level Neural Machine Translation: Methods and Evaluation

Cited by 9 publications

References 67 publications

Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models

Divide and Rule: Effective Pre-Training for Context-Aware Multi-Encoder Translation Models

SJTU-NICT's Supervised and Unsupervised Neural Machine Translation Systems for the WMT20 News Translation Task

N-ODE Transformer: A Depth-Adaptive Variant of the Transformer Using Neural Ordinary Differential Equations

Contact Info

Product

Resources

About