Investigating alignment interpretability for low-resource NMT

Abstract

The attention mechanism for Neural Machine Translation (NMT) added flexibility to neural models, and the possibility to visualize softalignments between source and target representations. While there is much debate about the impact of attention in the translation quality of neural models [25,40,35,32], in this paper we propose a different assessment, investigating soft-alignment interpretability in low-resource scenarios. We experiment with different architectures (RNN [5], 2D-CNN [15], and Transformer [36]), comparing their capacity to produce directly exploitable alignments. For evaluating exploitability, we replicate the Unsupervised Word Segmentation (UWS) task from Godard et al. [21]. There, source words are translated into unsegmented phone sequences. Posterior to training, the resulting soft-alignments are used for producing segmentation over the target side. Our results show that the RNN produces the most exploitable alignments in this scenario. We thus conclude by investigating methods for increasing its UWS scores. We compare the following methodologies: monolingual pre-training, input representation augmentation (hybrid model), and explicit word length optimization during training. We reach the best results by using the hybrid model, which uses an intermediate monolingual-rooted segmentation from a non-parametric Bayesian model [24] to enrich the input representation before training.