Exploration of Corpus Augmentation Approach for English-Hindi Bidirectional Statistical Machine Translation System

Jaya, K.; Gupta, Deepa

doi:10.11591/ijece.v6i3.8904

“…They analysed the effect of topic modeling on general corpus sentences mixed with in-domain text. Jaya and Gupta (2016) considered an Out-Of-Vocabulary (OOV) approach to standardize the dataset. They investigated a corpus augmentation method to improve the translation quality of the bidirectional English-Hindi SMT system.…”

Section: Related Workmentioning

confidence: 99%

Context-based Machine Translation of English-Hindi using CE-Encoder

Bansal¹,

Lobiyal²

2021

Journal of Computer Science

0

View full text Add to dashboard Cite

The difficulty in obtaining accurate word alignment and determining a target word that is the best candidate for a source context in machine translation leads to different translations. In this study, we propose a method with a more accurate context model. Our Neural Machine Translation (NMT) approach focuses on the encoder to apprehend the meaning of source sentences for improved translation. The recurrent encoder works by taking into consideration the history and future information of the source context. In this study, we implement the proposed approach into three steps. Firstly, we learn the representation of future context in advance. Secondly, a context-based recurrent encoder called as CE-Encoder with two-level Gated Recurrent Unit (GRU) is used. In this, the bottom-level GRU gathers history data of a sentence and top-level GRU assembles future data information. Finally, the future learned context and the history information from the opposite direction is integrated. The distinguishing factor of the proposed framework from the existing models, specifically Bidirectional Recurrent Neural Network (BiRNN) is that, the current models have not spent substantial time and capacity in learning future context or disambiguating source and target words based on the context which is defined by source sentence. We conduct experiments on the datasets from ILCC and CFILT for the English-Hindi language pair. From the comparative evaluation, we observed that the proposed model outperforms the Bidirectional RNN encoder in terms of translation quality. The proposed model has shown the improvement of 7 Bleu points using the ILCC dataset and 9 points using the CFILT dataset over BiRNN.

show abstract

“…They analysed the effect of topic modelling on general corpus sentences mixed with indomain text. Jaya and Gupta (2016) considered an Out-Of-Vocabulary (OOV) approach to standardize the dataset. They investigated a corpus augmentation method to improve the translation quality of the bidirectional English-Hindi SMT system.…”

Section: Related Workmentioning

confidence: 99%

Context-based Machine Translation of English-Hindi using CE-Encoder

Bansal¹,

Lobiyal²

2021

Journal of Computer Science

0

View full text Add to dashboard Cite

The difficulty in obtaining accurate word alignment and determining a target word that is the best candidate for a source context in machine translation leads to different translations. In this study, we propose a method with a more accurate context model. Our Neural Machine Translation (NMT) approach focuses on the encoder to apprehend the meaning of source sentences for improved translation. The recurrent encoder works by taking into consideration the history and future information of the source context. In this study, we implement the proposed approach into three steps. Firstly, we learn the representation of future context in advance. Secondly, a context-based recurrent encoder called as CE-Encoder with two-level Gated Recurrent Unit (GRU) is used. In this, the bottom-level GRU gathers history data of a sentence and top-level GRU assembles future data information. Finally, the future learned context and the history information from the opposite direction is integrated. The distinguishing factor of the proposed framework from the existing models, specifically Bidirectional Recurrent Neural Network (BiRNN) is that, the current models have not spent substantial time and capacity in learning future context or disambiguating source and target words based on the context which is defined by source sentence. We conduct experiments on the datasets from ILCC and CFILT for the English-Hindi language pair. From the comparative evaluation, we observed that the proposed model outperforms the Bidirectional RNN encoder in terms of translation quality. The proposed model has shown the improvement of 7 Bleu points using the ILCC dataset and 9 points using the CFILT dataset over BiRNN.

show abstract