Vocabulary Adaptation for Domain Adaptation in Neural Machine Translation

Abstract

Neural network methods exhibit strong performance only in a few resource-rich domains. Practitioners therefore employ domain adaptation from resource-rich domains that are, in most cases, distant from the target domain. Domain adaptation between distant domains (e.g., movie subtitles and research papers), however, cannot be performed effectively due to mismatches in vocabulary; it will encounter many domain-specific words (e.g., "angstrom") and words whose meanings shift across domains (e.g., "conductor"). In this study, aiming to solve these vocabulary mismatches in domain adaptation for neural machine translation (NMT), we propose vocabulary adaptation, a simple method for effective fine-tuning that adapts embedding layers in a given pretrained NMT model to the target domain. Prior to fine-tuning, our method replaces the embedding layers of the NMT model by projecting general word embeddings induced from monolingual data in a target domain onto a source-domain embedding space. Experimental results indicate that our method improves the performance of conventional fine-tuning by 3.86 and 3.28 BLEU points in En→Ja and De→En translation, respectively.

Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation

“…Lewis et al (2020) first replace the embedding layer with an independent encoder, of which vocabulary and parameters are learned from the downstream corpus. Along this line, Sato et al (2020) exploit external monolingual data to construct a new embedding layer and achieve improvements in domain adaptation. This series of studies empirically confirm the necessity of the suitable vocabulary for the finetuning stage.…”

“…Pretrain-finetune paradigm has been highly successful on tackling challenging problems in natural language processing, e.g., domain adaptation (Sato et al, 2020;Yao et al, 2020), incremental learning (Khayrallah et al, 2018;Wan et al, 2020), as well as knowledge transferring (Liu et al, 2020b). The rise of large-scale pre-trained language models further attracts increasing attention towards this strategy (Devlin et al, 2019;Edunov et al, 2019).…”

Robust Backed-off Estimation of Out-of-Vocabulary Embeddings

“…However, it has been reported that adversarial typos can degrade a BERT model that uses subword tokenization (Pruthi et al, 2019;Sun et al, 2020). Subword meanings change across domains, making domain adaptation difficult (Sato et al, 2020). These problems are more critical in the processing of noisy text (Wang et al, 2020;Niu et al, 2020).…”

“…The dynamic nature of language and the limited size of training data requires neural network models to handle out-of-vocabulary (OOV) words that are absent from the training data. We thus use an UNK embedding shared among diverse OOV words or break those OOV words into semanticallyambiguous subwords (even characters), leading to poor task performance (Peng et al, 2019;Sato et al, 2020).…”

Transitive Transfer Sparse Coding for Distant Domain

“…Distant domain transfer learning (DDTL) is a pretty common scenario in real transfer learning applications [1][2][3][4][5]. However, there is one key challenge to overcome, whereby the current transfer learning approaches will not work well when the source domain is very distant from the target domain.…”