Hunter NMT System for WMT18 Biomedical Translation Task: Transfer Learning in Neural Machine Translation

Abstract

This paper describes the submission of Hunter Neural Machine Translation (NMT) to the WMT'18 Biomedical translation task from English to French. The discrepancy between training and test data distribution brings a challenge to translate text in new domains. Beyond the previous work of combining in-domain with out-of-domain models, we found accuracy and efficiency gain in combining different in-domain models. We conduct extensive experiments on NMT with transfer learning. We train on different in-domain Biomedical datasets one after another. That means parameters of the previous training serve as the initialization of the next one. Together with a pre-trained out-of-domain News model, we enhanced translation quality with 3.73 BLEU points over the baseline. Furthermore, we applied ensemble learning on training models of intermediate epochs and achieved an improvement of 4.02 BLEU points over the baseline. Overall, our system is 11.29 BLEU points above the best system of last year on the EDP 2017 test set.

UCAM Biomedical Translation at WMT19: Transfer Learning Multi-domain Ensembles

“…Previous work on transfer learning typically aims to find a single model that performs well on a known domain of interest (Khan et al, 2018).…”

“…Transfer learning is an approach in which a model is trained using knowledge from an existing model (Khan et al, 2018). Transfer learning typically involves initial training on a large, general domain corpus, followed by fine-tuning on the domain of interest.…”

Exploring Transfer Learning and Domain Data Selection for the Biomedical Translation

“…The resultant BLEU scores did not improve more than 0.5. Khan et al (2018) trained three NMT systems with different corpus grouping. One experiment included only in-domain corpus, whereas two experiments were performed to train in-domain corpus by initializing the training parameters from general domain system.…”

“…To overcome this challenge various studies explore numerous techniques to improve NMT quality especially in low resource settings. Domain adaptation (Freitag and Al-Onaizan, 2016), transfer learning (Zoph et al, 2016;Khan et al, 2018), fine tuning (Dakwale and Monz, 2017;Huck et al, 2018) and data selective training (van der Wees et al, 2017); are few terms being interchangeably used for such techniques as reported in the literature.…”

Seq2Edits: Sequence Transduction Using Span-level Edit Operations

“…We report SARI scores as (re)computed byMallinson et al (2020) for all systems in Table8to ensure comparability.8 Multi-stage fine-tuning has been proven effective for other sequence tasks such as machine translation(Khan et al, 2018;Saunders et al, 2019).…”