Using Factored Word Representation in Neural Network Language Models

Niehues, Jan; Ha, Thanh-Le; Cho, Eunah; Waibel, Alex

doi:10.18653/v1/w16-2208

Cited by 13 publications

(10 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Applying more tightly coupled linguistic factors on the target for NMT has been previously investigated. Niehues et al (2016) proposed a factored RNN-based language model for re-scoring an n-best list produced by a phrase-based MT system. In recent work, Martínez et al (2016) implemented a factored NMT decoder which generated both lemmas and morphological tags.…”

Section: Related Workmentioning

confidence: 99%

Predicting Target Language CCG Supertags Improves Neural Machine Translation

Nădejde¹,

Reddy²,

Sennrich³

et al. 2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

Neural machine translation (NMT) models are able to partially learn syntactic information from sequential lexical information. Still, some complex syntactic phenomena such as prepositional phrase attachment are poorly modeled. This work aims to answer two questions: 1) Does explicitly modeling target language syntax help NMT? 2) Is tight integration of words and syntax better than multitask training? We introduce syntactic information in the form of CCG supertags in the decoder, by interleaving the target supertags with the word sequence. Our results on WMT data show that explicitly modeling targetsyntax improves machine translation quality for German→English, a high-resource pair, and for Romanian→English, a lowresource pair and also several syntactic phenomena including prepositional phrase attachment. Furthermore, a tight coupling of words and syntax improves translation quality more than multitask training. By combining target-syntax with adding source-side dependency labels in the embedding layer, we obtain a total improvement of 0.9 BLEU for German→English and 1.2 BLEU for Romanian→English.

show abstract

Section: Related Workmentioning

confidence: 99%

Predicting Target Language CCG Supertags Improves Neural Machine Translation

Nădejde¹,

Reddy²,

Sennrich³

et al. 2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

show abstract

“…2016a), and more in line with our proposed approach, pre-translation (Niehues et al . 2016). Recently, transformer-based models have outperformed the attention-based GRU models on the various benchmark datasets and have become the state-of-the-art technique in NMT.…”

Section: Results and Analysismentioning

confidence: 99%

Neural machine translation of low-resource languages using SMT phrase pair injection

et al. 2020

View full text Add to dashboard Cite

Neural machine translation (NMT) has recently shown promising results on publicly available benchmark datasets and is being rapidly adopted in various production systems. However, it requires high-quality large-scale parallel corpus, and it is not always possible to have sufficiently large corpus as it requires time, money, and professionals. Hence, many existing large-scale parallel corpus are limited to the specific languages and domains. In this paper, we propose an effective approach to improve an NMT system in low-resource scenario without using any additional data. Our approach aims at augmenting the original training data by means of parallel phrases extracted from the original training data itself using a statistical machine translation (SMT) system. Our proposed approach is based on the gated recurrent unit (GRU) and transformer networks. We choose the Hindi–English, Hindi–Bengali datasets for Health, Tourism, and Judicial (only for Hindi–English) domains. We train our NMT models for 10 translation directions, each using only 5–23k parallel sentences. Experiments show the improvements in the range of 1.38–15.36 BiLingual Evaluation Understudy points over the baseline systems. Experiments show that transformer models perform better than GRU models in low-resource scenarios. In addition to that, we also find that our proposed method outperforms SMT—which is known to work better than the neural models in low-resource scenarios—for some translation directions. In order to further show the effectiveness of our proposed model, we also employ our approach to another interesting NMT task, for example, old-to-modern English translation, using a tiny parallel corpus of only 2.7K sentences. For this task, we use publicly available old-modern English text which is approximately 1000 years old. Evaluation for this task shows significant improvement over the baseline NMT.

show abstract

“…Factored word representations have also been considered in neural language models (Niehues et al, 2016;Alexandrescu and Kirchhoff, 2006;Wu et al, 2012), and more recently in a neural machine translation architecture as input features (Sennrich and Haddow, 2016) and in the output by separating the lemma and morphological factors (García-Martínez et al, 2016). One contribution of the current paper is the investigation of new variants of the latter architecture.…”

Section: Related Workmentioning

confidence: 99%

Word Representations in Factored Neural Machine Translation

Burlot¹,

García-Martínez²,

Barrault³

et al. 2017

Proceedings of the Second Conference on Machine Translation

View full text Add to dashboard Cite

Translation into a morphologically rich language requires a large output vocabulary to model various morphological phenomena, which is a challenge for neural machine translation architectures. To address this issue, the present paper investigates the impact of having two output factors with a system able to generate separately two distinct representations of the target words. Within this framework, we investigate several word representations that correspond to different distributions of morpho-syntactic information across both factors. We report experiments for translation from English into two morphologically rich languages, Czech and Latvian, and show the importance of explicitly modeling target morphology.

show abstract

Using Factored Word Representation in Neural Network Language Models

Cited by 13 publications

References 16 publications

Predicting Target Language CCG Supertags Improves Neural Machine Translation

Predicting Target Language CCG Supertags Improves Neural Machine Translation

Neural machine translation of low-resource languages using SMT phrase pair injection

Word Representations in Factored Neural Machine Translation

Contact Info

Product

Resources

About