The IIT Bombay English-Hindi Parallel Corpus

Kunchukuttan, Anoop; Mehta, Pratik; Bhattacharyya, Pushpak

doi:10.48550/arxiv.1710.02855

Cited by 13 publications

(13 citation statements)

References 3 publications

(4 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the monolingual corpus, we extracted it from CC-100 according to ; . For bilingual corpus, we use the same corpus as INFOXLM (Chi et al, 2020b), including MultiUN (Ziemski et al, 2016), IIT Bombay (Kunchukuttan et al, 2017), OPUS (Tiedemann, 2012) and WikiMatrix . In order to balance the size of data for high-resource and lowresource languages, both monolingual and parallel corpora were sampled with parameter alpha of 0.1 referred to the method of (Lample and Conneau, 2019).…”

Section: Data and Modelmentioning

confidence: 99%

ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

Ouyang

Wang

Pang

et al. 2020

Preprint

View full text Add to dashboard Cite

Recent studies have demonstrated that pretrained cross-lingual models achieve impressive performance on downstream cross-lingual tasks. This improvement stems from the learning of a large amount of monolingual and parallel corpora. While it is generally acknowledged that parallel corpora are critical for improving the model performance, existing methods are often constrained by the size of parallel corpora, especially for the low-resource languages. In this paper, we propose ERNIE-M, a new training method that encourages the model to align the representation of multiple languages with monolingual corpora, to break the constraint of parallel corpus size on the model performance. Our key insight is to integrate the idea of back translation in the pretraining process. We generate pseudo-parallel sentences pairs on a monolingual corpus to enable the learning of semantic alignment between different languages, which enhances the semantic modeling of cross-lingual models. Experimental results show that ERNIE-M outperforms existing cross-lingual models and delivers new state-of-the-art results on various cross-lingual downstream tasks. The codes and pre-trained models will be made publicly available.

show abstract

Section: Data and Modelmentioning

confidence: 99%

ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

Ouyang

Wang

Pang

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…In our English to Hindi machine translation experiments, we have used the publicly available IIT Bombay (IITB) English-Hindi Parallel Corpus (Kunchukuttan et al, 2017). The training data in the IITB corpus consists of nearly 1.5M training samples.…”

Section: Dataset Detailsmentioning

confidence: 99%

Domain Adaptation of NMT models for English-Hindi Machine Translation Task at AdapMT ICON 2020

Joshi,

Karnavat,

Jirapure

et al. 2020

Preprint

View full text Add to dashboard Cite

Recent advancements in Neural MachineTranslation (NMT) models have proved to produce a state of the art results on machine translation for low resource Indian languages. This paper describes the neural machine translation systems for the English-Hindi language presented in AdapMT Shared Task ICON 2020. The shared task aims to build a translation system for Indian languages in specific domains like Artificial Intelligence (AI) and Chemistry using a small in-domain parallel corpus. We evaluated the effectiveness of two popular NMT models i.e, LSTM, and Transformer architectures for the English-Hindi machine translation task based on BLEU scores. We train these models primarily using the out of domain data and employ simple domain adaptation techniques based on the characteristics of the in-domain dataset. The fine-tuning and mixed-domain data approaches are used for domain adaptation. Our team was ranked first in the chemistry and general domain En-Hi translation task and second in the AI domain En-Hi translation task.

show abstract

“…from IITB (Kunchukuttan et al, 2017); English-Russian, English-Arabic, and English-Chinese parallel data from the UN Corpus (Ziemski et al, 2016); English-Tamil and English-Telugu from Wikimatrix (Schwenk et al, 2019). We report the counts in Table 1.…”

Section: Datasets and Preprocessingmentioning

confidence: 99%

Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages

García¹,

Siddhant²,

Fırat³

et al. 2020

Preprint

View full text Add to dashboard Cite

Unsupervised translation has reached impressive performance on resource-rich language pairs such as English-French and English-German. However, early studies have shown that in more realistic settings involving lowresource, rare languages, unsupervised translation performs poorly, achieving less than 3.0 BLEU. In this work, we show that multilinguality is critical to making unsupervised systems practical for low-resource settings. In particular, we present a single model for 5 low-resource languages (Gujarati, Kazakh, Nepali, Sinhala, and Turkish) to and from English directions, which leverages monolingual and auxiliary parallel data from other highresource language pairs via a three-stage training scheme. We outperform all current stateof-the-art unsupervised baselines for these languages, achieving gains of up to 14.4 BLEU. Additionally, we outperform a large collection of supervised WMT submissions for various language pairs as well as match the performance of the current state-of-the-art supervised model for NeÑEn. We conduct a series of ablation studies to establish the robustness of our model under different degrees of data quality, as well as to analyze the factors which led to the superior performance of the proposed approach over traditional unsupervised models.

show abstract

The IIT Bombay English-Hindi Parallel Corpus

Cited by 13 publications

References 3 publications

ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora

Domain Adaptation of NMT models for English-Hindi Machine Translation Task at AdapMT ICON 2020

Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages

Contact Info

Product

Resources

About