Neural Machine Translation for Low Resource Languages using
            Bilingual Lexicon Induced from Comparable Corpora

Ramesh, Sree Harsha; Sankaranarayanan, Krishna Prasad

doi:10.18653/v1/n18-4016

Cited by 16 publications

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, the Transformer suffers in low-resource translation tasks [7][8][9][10][11] where there is not a large-scale parallel corpus available. The core of this problem is the mismatch between the big model capacity and the small training parallel data available.…”

Section: Introductionmentioning

confidence: 99%

Boosting the Transformer with the BERT Supervision in Low-Resource Machine Translation

Yan

et al. 2022

Applied Sciences

View full text Add to dashboard Cite

Previous works trained the Transformer and its variants end-to-end and achieved remarkable translation performance when there are huge parallel sentences available. However, these models suffer from the data scarcity problem in low-resource machine translation tasks. To deal with the mismatch problem between the big model capacity of the Transformer and the small parallel training data set, this paper adds the BERT supervision on the latent representation between the encoder and the decoder of the Transformer and designs a multi-step training algorithm to boost the Transformer on such a basis. The algorithm includes three stages: (1) encoder training, (2) decoder training, and (3) joint optimization. We introduce the BERT of the target language in the encoder and the decoder training and alleviate the data starvation problem of the Transformer. After the training stage, the BERT will not further attend the inference section explicitly. Another merit of our training algorithm is that it can further enhance the Transformer in the task where there are limited parallel sentence pairs but large amounts of monolingual corpus of the target language. The evaluation results on six low-resource translation tasks suggest that the Transformer trained by our algorithm significantly outperforms the baselines which were trained end-to-end in previous works.

show abstract

Section: Introductionmentioning

confidence: 99%