Incorporating Discrete Translation Lexicons into Neural Machine Translation

Abstract

Neural machine translation (NMT) often makes mistakes in translating low-frequency content words that are essential to understanding the meaning of the sentence. We propose a method to alleviate this problem by augmenting NMT systems with discrete translation lexicons that efficiently encode translations of these low-frequency words. We describe a method to calculate the lexicon probability of the next word in the translation candidate by using the attention vector of the NMT model to select which source word lexical probabilities the model should focus on. We test two methods to combine this probability with the standard NMT probability: (1) using it as a bias, and (2) linear interpolation. Experiments on two corpora show an improvement of 2.0-2.3 BLEU and 0.13-0.44 NIST score, and faster convergence time.

Six Challenges for Neural Machine Translation

“…For instance, prior work used the alignments provided by the attention model to interpolate word translation decisions with traditional probabilistic dictionaries (Arthur et al, 2016), for the introduction of coverage and fertility models (Tu et al, 2016), etc. But is the attention model in fact the proper means? To examine this, we compare the soft alignment matrix (the sequence of attention vectors) with word alignments obtained by traditional word alignment methods.…”

Jointly Learning to Align and Translate with Transformer Models

“…A straightforward application of word alignments is to generate bilingual lexica from parallel corpora. Word alignments have also been used for external dictionary assisted translation (Chatterjee et al, 2017;Alkhouli et al, 2018;Arthur et al, 2016) to improve translation of low frequency words or to comply with certain terminology guidelines. Documents and webpages often contain word annotations such as formatting styles and hyperlinks, which need to be preserved in the translation.…”

Prior Knowledge Integration for Neural Machine Translation using Posterior Regularization

“…Our work is closely related to recent work on injecting prior knowledge into NMT (Arthur et al, 2016;Cohn et al, 2016;Tang et al, 2016;Feng et al, 2016;. The major difference is that our approach aims to provide a general framework for incorporating arbitrary prior knowledge sources while keeping the neural translation model unchanged.…”

“…It is natural to leverage a bilingual dictionary D to improve neural machine translation. Arthur et al (2016) propose to incorporate discrete translation lexicons into NMT by using the attention vector to select lexical probabilities on which to be focused.…”