Effective preprocessing based neural machine translation for English to Telugu cross-language information retrieval

Abstract

<span id="docs-internal-guid-5b69f940-7fff-f443-1f09-a00e5e983714"><span>In cross-language information retrieval (CLIR), the neural machine translation (NMT) plays a vital role. CLIR retrieves the information written in a language which is different from the user's query language. In CLIR, the main concern is to translate the user query from the source language to the target language. NMT is useful for translating the data from one language to another. NMT has better accuracy for different languages like English to German and so-on. In this paper, NMT has applied for translating English to Indian languages, especially for Telugu. Besides NMT, an effort is also made to improve accuracy by applying effective preprocessing mechanism. The role of effective preprocessing in improving accuracy will be less but countable. Machine translation (MT) is a data-driven approach where parallel corpus will act as input in MT. NMT requires a massive amount of parallel corpus for performing the translation. Building an English - Telugu parallel corpus is costly because they are resource-poor languages. Different mechanisms are available for preparing the parallel corpus. The major issue in preparing parallel corpus is data replication that is handled during preprocessing. The other issue in machine translation is the out-of-vocabulary (OOV) problem. Earlier dictionaries are used to handle OOV problems. To overcome this problem the rare words are segmented into sequences of subwords during preprocessing. The parameters like accuracy, perplexity, cross-entropy and BLEU scores shows better translation quality for NMT with effective preprocessing.</span></span>

BiLSTMs and BPE for English to Telugu CLIR

“…However, in the context of the OOV problem, the efficiency of NMT may fall as the quantity of unfamiliar phrases grows. Despite the utilization of a BPE technique by B N V Narasimha Raju et al [13] to tackle certain OOV concerns, there are still hurdles in translation production. BPE has been used for parallel corpora by Mattia A.…”

“…Sentences with a high frequency of words may result in OOV issues if resources are limited [16]- [18]. As a method of data compression, BPE [7], [13] is useful for merging byte pairs that appear frequently. The BPE mechanism may be helpful in word segmentation.…”

Lexical Based Reordering Models for English to Telugu Machine Translation

“…NMT needs a huge volume of parallel corpus to do the translations. Telugu and English are resource-constrained languages, creating a parallel corpus is expensive [16].…”

“…According the previous works [9][10][11][12][13][14][15][16][17], the translation among one language to other languages are chosen based on leading languages in top countries using MT. These translations are very effective in terms of standards and quality.…”