A Discriminative Neural Model for Cross-Lingual Word Alignment

Abstract

We introduce a novel discriminative word alignment model, which we integrate into a Transformer-based machine translation model. In experiments based on a small number of labeled examples (∼1.7K-5K sentences) we evaluate its performance intrinsically on both English-Chinese and English-Arabic alignment, where we achieve major improvements over unsupervised baselines (11-27 F1). We evaluate the model extrinsically on data projection for Chinese NER, showing that our alignments lead to higher performance when used to project NER tags from English to Chinese. Finally, we perform an ablation analysis and an annotation experiment that jointly support the utility and feasibility of future manual alignment elicitation.

SimAlign: High Quality Word Alignments Without Parallel Training Data Using Static and Contextualized Embeddings

“…Li et al (2019) propose two methods to extract alignments from NMT models, however they do not outperform fast-align. Stengel-Eskin et al (2019) compute similarity matrices of encoder-decoder representations that are leveraged for word alignments, together with supervised learning, which requires manually annotated alignment. We find our proposed methods to be competitive with these approaches.…”

A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT

“…It formalizes word alignment as a collection of SQuAD-style span prediction problems (Rajpurkar et al, 2016) and solves them with multilingual BERT (Devlin et al, 2019). We experimentally show that our proposed model significantly outperformed both (Garg et al, 2019) and (Stengel-Eskin et al, 2019).…”

“…Most previous works that use them for word alignment (Yang et al, 2013;Tamura et al, 2014;Legrand et al, 2016) achieved accuracies that are basically comparable to GIZA++. However, the accuracy of recent works (Garg et al, 2019;Stengel-Eskin et al, 2019;Zenkel et al, 2020) based on the Transformer (Vaswani et al, 2017), which is the state-of-the art neural machine translation model, have started to outperform GIZA++. Garg et al (2019) made the attention of the Transformer more closely resembled the word alignment, and achieved better accuracy than GIZA++ when they used alignments obtained from it for supervision.…”

“…Stengel-Eskin et al (2019) proposed a supervised word alignment method using the hidden states of the Transformer, and significantly outperformed FastAlign (11-27 F1 points) using a small number of gold word alignments (1.7K-5K sentences). However, both Garg et al (2019) and Stengel-Eskin et al (2019) required more than a million parallel sentences to pretrain their models. Applying these methods to low-resource language pairs and domains is difficult.…”

“…Our main contribution is that we make supervised word alignment more practical. Stengel-Eskin et al (2019) argued that supervised word alignment is a viable option. They concluded that alignment annotation could be performed rapidly 4.4 sentences per minute by annotators with minimal experience using a web-based crowd-sourcing interface.…”