Framework for Handling Rare Word Problems in Neural Machine Translation System Using Multi-Word Expressions

Abstract

Neural machine translation (NMT) is an ongoing technique used to implement machine translation (MT) systems. Natural language processing (NLP) researchers have shown that NMT systems are unable to deal with out-of-vocabulary (OOV) words and multi-word expressions (MWEs) in the text. OOV words are those that are not part of the current vocabulary of the NMT system. MWEs are phrases that consist of a minimum of two terms but are treated as a single unit. MWEs have great importance in NLP, linguistic theory, and MT systems. In this article, OOV words and MWEs are handled for the Punjabi to English NMT system. A parallel corpus for Punjabi to English containing MWEs was developed and used to train the different models of NMT. Punjabi is a low-resource language as it lacks the availability of a large parallel corpus for building various NLP tools, and this is an attempt to improve the accuracy of Punjabi in the English NMT system by using named entities and MWEs in the corpus. The developed NMT models were assessed using human evaluation through adequacy and fluency as well as automated assessment tools such as the bilingual evaluation study (BLEU) and translation error rate (TER) score. Results show that using word embedding (WE) and MWEs corpus increased the accuracy of translation for the Punjabi to English language pair. The best BLEU score obtained was 15.45 for the small test set, 43.32 for the medium test set, and 34.5 for the large test set, respectively. The best TER rate score obtained was 57.34% for the small test set, 37.29% for the medium test set, and 53.79% for the large test set, repectively.

Effective Spam Detection with Machine Learning

“…For example, computational intensity becomes more and more relevant as the volume of data grows, and (re)training cost is important for an agile startup that wants to iterate quickly to improve time to market. Future research should also compare computational intensity scales and the training cost of each model (Garg et al, 2022). In addition, as machine learning models are increasingly being used in various domains (Tembhurne et al, 2022), it is important to investigate their ethical implications (Konagala & Bano, 2020).…”

Analysis on the Legal System of International Technology Trade Management Based on Data Mining Analysis

“…However, existing research faces several issues. At present, no specialized economic theory can directly explain and demonstrate the impact of human networks on the structure of international trade networks (Garg et al, 2022). The existing research mainly uses trade flow data to construct international trade networks directly, ignoring the differences in relative trade intensity and failing to reflect the different impacts and roles of trade flow on two countries with trade relations (Garg et al, 2022).…”

“…At present, no specialized economic theory can directly explain and demonstrate the impact of human networks on the structure of international trade networks (Garg et al, 2022). The existing research mainly uses trade flow data to construct international trade networks directly, ignoring the differences in relative trade intensity and failing to reflect the different impacts and roles of trade flow on two countries with trade relations (Garg et al, 2022). Therefore, this article constructs a new framework for studying international trade networks, dividing them into different levels based on the flow, proportion, and preferences of import and export trade between countries (Hamza et al, 2022).…”