Generalizing Back-Translation in Neural Machine Translation

Graça, Miguel; Kim, Yunsu; Schamper, Julian; Khadivi, Shahram; Ney, Hermann

doi:10.18653/v1/w19-5205

Cited by 27 publications

(22 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Back-translation is simple and easy to achieve without modifying the architecture of the machine translation models. Back-translation has been studied in both SMT [111][112][113] and NMT [23,80,110,[114][115][116].…”

Section: Pivot Translationmentioning

confidence: 99%

A Survey of Orthographic Information in Machine Translation

et al. 2021

View full text Add to dashboard Cite

Machine translation is one of the applications of natural language processing which has been explored in different languages. Recently researchers started paying attention towards machine translation for resource-poor languages and closely related languages. A widespread and underlying problem for these machine translation systems is the linguistic difference and variation in orthographic conventions which causes many issues to traditional approaches. Two languages written in two different orthographies are not easily comparable but orthographic information can also be used to improve the machine translation system. This article offers a survey of research regarding orthography’s influence on machine translation of under-resourced languages. It introduces under-resourced languages in terms of machine translation and how orthographic information can be utilised to improve machine translation. We describe previous work in this area, discussing what underlying assumptions were made, and showing how orthographic knowledge improves the performance of machine translation of under-resourced languages. We discuss different types of machine translation and demonstrate a recent trend that seeks to link orthographic information with well-established machine translation methods. Considerable attention is given to current efforts using cognate information at different levels of machine translation and the lessons that can be drawn from this. Additionally, multilingual neural machine translation of closely related languages is given a particular focus in this survey. This article ends with a discussion of the way forward in machine translation with orthographic information, focusing on multilingual settings and bilingual lexicon induction.

show abstract

Section: Pivot Translationmentioning

confidence: 99%

A Survey of Orthographic Information in Machine Translation

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Various studies have investigated back-translation to improve the backward model, to select the most suitable generation/decoding methods for generating the synthetic data and to reduce the impact of higher ratio of the synthetic to the authentic bitext. The quality of the models trained using back-translation depends on the quality of the backward model [16,17,22,5,19,29,52]. To improve the quality of the synthetic parallel data, [22] used iterative back-translation -iteratively using the back-translated data to improve both the backward and forward models.…”

Section: Leveraging Monolingual Data For Nmtmentioning

confidence: 99%

Tag-less Back-Translation

Abdulmumin

Galadanci

Aliyu

2021

Preprint

View full text Add to dashboard Cite

An effective method to generate a large number of parallel sentences for training improved neural machine translation (NMT) systems is the use of the back-translations of the target-side monolingual data. The standard back-translation method has been shown to be unable to efficiently utilize the available huge amount of existing monolingual data because of the inability of translation models to differentiate between the authentic and synthetic parallel data during training. Tagging, or using gates, has been used to enable translation models to distinguish between synthetic and authentic data, improving standard back-translation and also enabling the use of iterative back-translation on language pairs that underperformed using standard back-translation. In this work, we approach back-translation as a domain adaptation problem, eliminating the need for explicit tagging. In the approach - tag-less back-translation - the synthetic and authentic parallel data are treated as out-of-domain and in-domain data respectively and, through pre-training and fine-tuning, the translation model is shown to be able to learn more efficiently from them during training. Experimental results have shown that the approach outperforms the standard and tagged back-translation approaches on low resource English-Vietnamese and English-German neural machine translation.

show abstract

“…While a separate target-to-source back-translation model is commonly used, how exactly the artificial source-side texts should be generated largely remains a yet-to-resolve research question. The sampling vs. search comparisons are already made in several works Imamura et al, 2018;Graça et al, 2019), while Wang et al (2019) being a most recent work introducing uncertainty-based confidence estimation as an alternative.…”

Section: Related Workmentioning

confidence: 99%

“…To build a competitive neural translation model, back-translation is a commonly used method (Bojar et al, 2018). Following recent studies, sampling from the back-translation model instead of using beamsearch is a popular alternative to obtain the synthetic source side Imamura et al, 2018;Graça et al, 2019).…”

Section: Smoothing In Back-translationmentioning

confidence: 99%

See 1 more Smart Citation

Unifying Input and Output Smoothing in Neural Machine Translation

Gao

Liao²,

Ney

2020

Proceedings of the 28th International Conference on Computational Linguistics

Self Cite

View full text Add to dashboard Cite

Soft contextualized data augmentation is a recent method that replaces one-hot representation of words with soft posterior distributions of an external language model, smoothing the input of neural machine translation systems. Label smoothing is another effective method that penalizes over-confident model outputs by discounting some probability mass from the true target word, smoothing the output of neural machine translation systems. Having the benefit of updating all word vectors in each optimization step and better regularizing the models, the two smoothing methods are shown to bring significant improvements in translation performance. In this work, we study how to best combine the methods and stack the improvements. Specifically, we vary the prior distributions to smooth with, the hyperparameters that control the smoothing strength, and the token selection procedures. We conduct extensive experiments on small datasets, evaluate the recipes on larger datasets, and examine the implications when back-translation is further used. Our results confirm cumulative improvements when input and output smoothing are used in combination, giving up to +1.9 BLEU scores on standard machine translation tasks and reveal reasons why these smoothing methods should be preferred.

show abstract

Generalizing Back-Translation in Neural Machine Translation

Cited by 27 publications

References 15 publications

A Survey of Orthographic Information in Machine Translation

A Survey of Orthographic Information in Machine Translation

Tag-less Back-Translation

Unifying Input and Output Smoothing in Neural Machine Translation

Contact Info

Product

Resources

About