UPC: An Open Word-Sense Annotated Parallel Corpora for Machine Translation Study

Abstract

Machine translation (MT) has recently attracted much research on various advanced techniques (i.e., statistical-based and deep learning-based) and achieved great results for popular languages. However, the research on it involving low-resource languages such as Korean often suffer from the lack of openly available bilingual language resources. In this research, we built the open extensive parallel corpora for training MT models, named Ulsan parallel corpora (UPC). Currently, UPC contains two parallel corpora consisting of Korean-English and Korean-Vietnamese datasets. The Korean-English dataset has over 969 thousand sentence pairs, and the Korean-Vietnamese parallel corpus consists of over 412 thousand sentence pairs. Furthermore, the high rate of homographs of Korean causes an ambiguous word issue in MT. To address this problem, we developed a powerful word-sense annotation system based on a combination of sub-word conditional probability and knowledge-based methods, named UTagger. We applied UTagger to UPC and used these corpora to train both statistical-based and deep learning-based neural MT systems. The experimental results demonstrated that using UPC, high-quality MT systems (in terms of the Bi-Lingual Evaluation Understudy (BLEU) and Translation Error Rate (TER) score) can be built. Both UPC and UTagger are available for free download and usage.

Word Sense Disambiguation Using Clustered Sense Labels

“…However, KorLex and Postech's WordNet contain only some part of the whole senses in Standard Korean Language Dictionary; KorLex contains only the senses that exist in English WordNet, and Postech's WordNet is partially translated from English WordNet. UWordMap has included most of sense vocabulary of Standard Korean Language Dictionary more recently, which is quite later than the sense annotated corpus release [42].…”

Word Sense Disambiguation Using Clustered Sense Labels

“…However, KorLex and Postech's WordNet contain only some part of the whole senses in Standard Korean Language Dictionary; KorLex contains only the senses that exist in English WordNet, and Postech's WordNet is partially translated from English WordNet. UWordMap has included most of sense vocabulary of Standard Korean Language Dictionary more recently, which is quite later than the sense annotated corpus release [42].…”

Improving the Performance of Vietnamese–Korean Neural Machine Translation with Contextual Embedding

“…To evaluate the accuracy of the system, we used over 412,000 Korean-Vietnamese sentence pairs that were published in our previous research [25]. In Vietnamese, we applied the word segmentation tool for sentences.…”

“…For Korean-Vietnamese machine translation (MT), Vu et al [25] published a dataset of more than 412,000 sentence pairs for training with the NMT model. Nguyen et al [26] performed a series of experiments with NMT by applying morphological analysis (MA) and word-sense disambiguation (WSD) to Korean sentences, before feeding language pairs into the MT system.…”

Analysis of Feature Extraction Model for Machine Korean Translation Judging System

“…The research will give a qualitative meta-analysis of published studies and government documentation on the use of MT in these two sectors. In the study [14], they created the Ulsan Parallel Corpora, an open, comprehensive parallel corpus for training MT models. There are over 969 thousand sentence pairings in the Korean-English database, and more than 412 thousand in the Korean-Vietnamese text corpus.…”

Word Sense Disambiguation Using Clustered Sense Labels

“…However, KorLex and Postech's WordNet contain only some part of the whole senses in Standard Korean Language Dictionary; KorLex contains only the senses that exist in English WordNet, and Postech's WordNet is partially translated from English WordNet. UWordMap has included most of sense vocabulary of Standard Korean Language Dictionary more recently, which is quite later than the sense annotated corpus release [42].…”

Improving the Performance of Vietnamese–Korean Neural Machine Translation with Contextual Embedding

“…To evaluate the accuracy of the system, we used over 412,000 Korean-Vietnamese sentence pairs that were published in our previous research [25]. In Vietnamese, we applied the word segmentation tool for sentences.…”

“…For Korean-Vietnamese machine translation (MT), Vu et al [25] published a dataset of more than 412,000 sentence pairs for training with the NMT model. Nguyen et al [26] performed a series of experiments with NMT by applying morphological analysis (MA) and word-sense disambiguation (WSD) to Korean sentences, before feeding language pairs into the MT system.…”

Analysis of Feature Extraction Model for Machine Korean Translation Judging System

“…The research will give a qualitative meta-analysis of published studies and government documentation on the use of MT in these two sectors. In the study [14], they created the Ulsan Parallel Corpora, an open, comprehensive parallel corpus for training MT models. There are over 969 thousand sentence pairings in the Korean-English database, and more than 412 thousand in the Korean-Vietnamese text corpus.…”

Word Sense Disambiguation Using Clustered Sense Labels

“…However, KorLex and Postech's WordNet contain only some part of the whole senses in Standard Korean Language Dictionary; KorLex contains only the senses that exist in English WordNet, and Postech's WordNet is partially translated from English WordNet. UWordMap has included most of sense vocabulary of Standard Korean Language Dictionary more recently, which is quite later than the sense annotated corpus release [42].…”

Improving the Performance of Vietnamese–Korean Neural Machine Translation with Contextual Embedding

“…To evaluate the accuracy of the system, we used over 412,000 Korean-Vietnamese sentence pairs that were published in our previous research [25]. In Vietnamese, we applied the word segmentation tool for sentences.…”

“…For Korean-Vietnamese machine translation (MT), Vu et al [25] published a dataset of more than 412,000 sentence pairs for training with the NMT model. Nguyen et al [26] performed a series of experiments with NMT by applying morphological analysis (MA) and word-sense disambiguation (WSD) to Korean sentences, before feeding language pairs into the MT system.…”

Analysis of Feature Extraction Model for Machine Korean Translation Judging System

“…The research will give a qualitative meta-analysis of published studies and government documentation on the use of MT in these two sectors. In the study [14], they created the Ulsan Parallel Corpora, an open, comprehensive parallel corpus for training MT models. There are over 969 thousand sentence pairings in the Korean-English database, and more than 412 thousand in the Korean-Vietnamese text corpus.…”