Jointly Learning to Align and Translate with Transformer Models

Abstract

The state of the art in machine translation (MT) is governed by neural approaches, which typically provide superior translation accuracy over statistical approaches. However, on the closely related task of word alignment, traditional statistical word alignment models often remain the go-to solution. In this paper, we present an approach to train a Transformer model to produce both accurate translations and alignments. We extract discrete alignments from the attention probabilities learnt during regular neural machine translation model training and leverage them in a multi-task framework to optimize towards translation and alignment objectives. We demonstrate that our approach produces competitive results compared to GIZA++ trained IBM alignment models without sacrificing translation accuracy and outperforms previous attempts on Transformer model based word alignment. Finally, by incorporating IBM model alignments into our multi-task training, we report significantly better alignment accuracies compared to GIZA++ on three publicly available data sets. Our implementation has been open-sourced 1 .

Self-Attention with Cross-Lingual Position Representation

“…We explore this hypothesis on the widely used Gold Alignment dataset 3 and follow Tang et al (2019) to perform the alignment. The only difference being that we average the attention matrices across all heads from the penultimate layer (Garg et al, 2019). The alignment error rate (AER, Och and Ney 2003), precision (P) and recall (R) are reported as the evaluation metrics.…”

Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

“…A closely related research area attempts to guide the attention mechanism, e.g. by incorporating alignment objectives (Garg et al, 2019), or improving the representation through external information such as syntactic supervision (Pham et al, 2019;Currey and Heafield, 2019;Deguchi et al, 2019). The third line of research argues that Transformer networks are over-parametrized and learn redundant information that can be pruned in various ways (Sanh et al, 2019).…”

Accurate Word Alignment Induction from Neural Machine Translation

“…The attention mechanism in NMT does not functionally play the role of word alignments between the source and the target, at least not in the same way as its analog in SMT. It is hard to interpret the attention activations and extract meaningful word alignments especially from Transformer (Garg et al, 2019). As a result, the most widely used word alignment tools are still external statistical models such as FAST-ALIGN (Dyer et al, 2013) and GIZA++ (Brown et al, 1993;Och and Ney, 2003).…”

“…1. However, such schedule only captures noisy word alignments (Ding et al, 2019;Garg et al, 2019). One of the major problems is that it induces alignment before observing the to-be-aligned target token (Peter et al, 2017;Ding et al, 2019).…”

“…To alleviate this problem, some researchers modify the transformer architecture by adding alignment modules that predict the to-be-aligned target token (Zenkel et al, 2019(Zenkel et al, , 2020 or modify the training loss by designing an alignment loss computed with full target sentence (Garg et al, 2019;Zenkel et al, 2020). Others argue that using only attention weights is insufficient for generating clean word alignment and propose to induce alignments with feature importance measures, such as leaveone-out measures (Li et al, 2019) and gradientbased measures (Ding et al, 2019).…”