Jointly Learning to Align and Translate with Transformer Models

Garg, Sarthak; Peitz, Stephan; Nallasamy, Udhyakumar; Paulik, Matthias

doi:10.48550/arxiv.1909.02074

Cited by 4 publications

(14 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the probability distribution given by the attention mechanism may not necessarily allow for inference on word alignment between vocabularies [10], thus requiring the use of statistical approaches for word alignment [2]. To address this problem, Garg et al [6] proposed to train a Transformer in a multi-task framework. Their multi-task loss function combines a negative log-likelihood loss of the Transformer (relative to the token prediction based on past tokens) with a conditional cross-entropy loss defined by the Kullback-Leibler (KL) divergence.…”

Section: Neural Machine Translationmentioning

confidence: 99%

“…In their approach, they used the attention probabilities from the penultimate attention head layer as labels for their supervised algorithm, thus dispensing the need for an annotated word alignment. The divergence between the attention probabilities of one arbitrarily chosen alignment head and the labeled alignment distribution is minimized as a KL divergence optimization problem [6].…”

Section: Neural Machine Translationmentioning

confidence: 99%

“…4.1). The training on the Europarl-v7 dataset [9] was performed with 100k sentences and the training on Wikipedia was performed one the embedding provided by FastText [ We compare our method to three state-of-the-art word alignment approaches: unsupervised Gromov-Wasserstein alignment [2], the neural-based Joint Learning alignment [6], and a supervised embedding alignment. We used a supervised alignment method that is based on a Transformer model, such as described in section 4.1, with a supervised word pair loss that is calculated between the embeddings of a set of known pair of "landmarks".…”

Section: Word Alignment Evaluationmentioning

confidence: 99%

“…Consequently, word translation can be achieved by finding a mapping function between the word embedding of two languages [8]. This can be done by either learning a transformation [12,2,5] or learning two embeddings that are aligned [6], i.e. are part of the same space.…”

Section: Introductionmentioning

confidence: 99%

“…are part of the same space. This has been done in supervised [12], semi-supervised [6,5], and unsupervised [2] manners. Although several methods for word translation have been proposed, most require large numbers of provided word pairs (supervised) to achieve a good performance [12,5], or (for unsupervised methods) may lack sufficient references to achieve a good quality alignment [2].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Learning aligned embeddings for semi-supervised word translation using Maximum Mean Discrepancy

Fonseca¹,

Dijk²

2020

Preprint

View full text Add to dashboard Cite

Word translation is an integral part of language translation. In machine translation, each language is considered a domain with its own word embedding. The alignment between word embeddings allows linking semantically equivalent words in multilingual contexts. Moreover, it offers a way to infer cross-lingual meaning for words without a direct translation. Current methods for word embedding alignment are either supervised, i.e. they require known word pairs, or learn a cross-domain transformation on fixed embeddings in an unsupervised way. Here we propose an end-to-end approach for word embedding alignment that does not require known word pairs. Our method, termed Word Alignment through MMD (WAM), learns embeddings that are aligned during sentence translation training using a localized Maximum Mean Discrepancy (MMD) constraint between the embeddings. We show that our method not only out-performs unsupervised methods, but also supervised methods that train on known word translations.Preprint. Under review.

show abstract

Section: Neural Machine Translationmentioning

confidence: 99%

Section: Neural Machine Translationmentioning

confidence: 99%

Section: Word Alignment Evaluationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Learning aligned embeddings for semi-supervised word translation using Maximum Mean Discrepancy

Fonseca¹,

Dijk²

2020

Preprint

View full text Add to dashboard Cite

show abstract

Spectral–Temporal Fusion of Satellite Images via an End-to-End Two-Stream Attention With an Effective Reconstruction Network

Benzenati

Kessentini

Kallel

2023

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

Due to technical and budget constraints on current optical satellites, the acquisition of satellite images with the best resolutions is not practicable. In this paper, aiming to produce products with high spectral (HS) and temporal (HT) resolutions, we introduced a two-stream spectral-temporal fusion technique based on attention mechanism called STA-Net. STA-Net aims to combine high spectral and low temporal (HSLT) resolution images with low spectral and high temporal (LSHT) resolution images to generate products with the best characteristics. The proposed technique involves two stages. In the first one, two fused images are generated by a two-stream architecture based on residual attention blocks. The Temporal Difference Estimator (TDE) stream estimates the temporal difference between HS images at desired and neighboring dates. The Reflectance Difference Estimator (RDE) is the second stream. It predicts the reflectance difference between the input images (HS-LS) to map LS images into HS products. In the second stage, a reconstruction network combines the latter two-stream outputs via an effective learnable weighted-sum strategy. The two-stage model is trained in an endto-end fashion using an effective loss function to ensure the best fusion quality. To the best of our knowledge, this work represents the first attempt to address the spectral-temporal fusion using an end-to-end deep neural network model. Experimental results conducted on two actual datasets of Sentinel-2 (HSLT:10 spectral bands and long revisit period) and Planetscope (LSHT: four spectral bands and daily images) images proved the effectiveness of the proposed technique with respect to baseline technique.

show abstract

Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation

Yang¹,

Bapna²,

Fırat³

et al. 2022

Preprint

View full text Add to dashboard Cite

Multilingual neural machine translation models are trained to maximize the likelihood of a mix of examples drawn from multiple language pairs. The dominant inductive bias applied to these models is a shared vocabulary and a shared set of parameters across languages; the inputs and labels corresponding to examples drawn from different language pairs might still reside in distinct subspaces. In this paper, we introduce multilingual crossover encoder-decoder (mXEncDec) to fuse language pairs at an instance level. Our approach interpolates instances from different language pairs into joint 'crossover examples' in order to encourage sharing input and output spaces across languages. To ensure better fusion of examples in multilingual settings, we propose several techniques to improve example interpolation across dissimilar languages under heavy data imbalance. Experiments on a large-scale WMT multilingual dataset demonstrate that our approach significantly improves quality on English-to-Many, Many-to-English and zero-shot translation tasks (from +0.5 BLEU up to +5.5 BLEU points). Results on code-switching sets demonstrate the capability of our approach to improve model generalization to out-of-distribution multilingual examples. We also conduct qualitative and quantitative representation comparisons to analyze the advantages of our approach at the representation level.

show abstract

Jointly Learning to Align and Translate with Transformer Models

Cited by 4 publications

References 0 publications

Learning aligned embeddings for semi-supervised word translation using Maximum Mean Discrepancy

Learning aligned embeddings for semi-supervised word translation using Maximum Mean Discrepancy

Spectral–Temporal Fusion of Satellite Images via an End-to-End Two-Stream Attention With an Effective Reconstruction Network

Multilingual Mix: Example Interpolation Improves Multilingual Neural Machine Translation

Contact Info

Product

Resources

About