Character Decomposition for Japanese-Chinese Character-Level Neural Machine Translation

Zhang, Jinyi; Matsumoto, Tadahiro

doi:10.1109/ialp48816.2019.9037677

Cited by 6 publications

(4 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…BPE denotes a subword based NMT with a joint vocabulary of 32K BPE tokens † † . Different to NMT for alphabetic languages, we can observe that the BPE approach in Japanese-Chinese NMT is inferior to the baseline using characters, which is consistent with the conclusions in [12], [13]. One reason is that the words in logographic languages are far shorter than words in alphabetic languages, which causes BPE failing to solve the low frequency words' problem.…”

Section: Resultssupporting

confidence: 77%

Sense-Aware Decoder for Character Based Japanese-Chinese NMT

LI,

REN

2024

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

Compared to subword based Neural Machine Translation (NMT), character based NMT eschews linguistic-motivated segmentation which performs directly on the raw character sequence, following a more absolute end-to-end manner. This property is more fascinating for machine translation (MT) between Japanese and Chinese, both of which use consecutive logographic characters without explicit word boundaries. However, there is still one disadvantage which should be addressed, that is, character is a less meaning-bearing unit than the subword, which requires the character models to be capable of sense discrimination. Specifically, there are two types of sense ambiguities existing in the source and target language, separately. With the former, it has been partially solved by the deep encoder and several existing works. But with the later, interestingly, the ambiguity in the target side is rarely discussed. To address this problem, we propose two simple yet effective methods, including a non-parametric pre-clustering for sense induction and a joint model to perform sense discrimination and NMT training simultaneously. Extensive experiments on Japanese↔Chinese MT show that our proposed methods consistently outperform the strong baselines, and verify the effectiveness of using sense-discriminated representation for character based NMT.

show abstract

Section: Resultssupporting

confidence: 77%

Sense-Aware Decoder for Character Based Japanese-Chinese NMT

LI,

REN

2024

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

show abstract

“…For Japanese-Chinese translation, Zhang et al proposed the following three data augmentation methods to improve the quality of Japanese-Chinese NMT: (1) radicals as an additional input feature [25]; (2) the created Chinese character decomposition table [26]; (3) a corpus augmentation approach [27], considering the lack of resources in bilingual corpora.…”

Section: Guokun Et Al Automatically Built a Corpus By Crawling Langua...mentioning

confidence: 99%

“…Corpus linguistics [3] Japanese-Chinese bilingual corpora [1], TED talks, [4,5] Web-crawled corpora [7][8][9][10][11]13,14,16,17,19,24] Other corpora [6,12,18,[20][21][22] Corpus augmentation [15,23,[25][26][27] The above related research showed that corpora play an important role in improving translation accuracy and in other directions of language processing. Thus, the construction of a Japanese-Chinese bilingual corpus for NMT has significant implications for the resource scarcity problem.…”

Section: Classification Related Workmentioning

confidence: 99%

WCC-JC: A Web-Crawled Corpus for Japanese-Chinese Neural Machine Translation

Zhang

Tian²,

Mao

et al. 2022

Applied Sciences

Self Cite

View full text Add to dashboard Cite

Currently, there are only a limited number of Japanese–Chinese bilingual corpora of a sufficient amount that can be used as training data for neural machine translation (NMT). In particular, there are few corpora that include spoken language such as daily conversation. In this research, we attempt to construct a Japanese–Chinese bilingual corpus of a certain scale by crawling the subtitle data of movies and TV series from the websites. We calculated the BLEU scores of the constructed WCC-JC (Web Crawled Corpus—Japanese and Chinese) and the other compared corpora. We also manually evaluated the translation results using the translation model trained on the WCC-JC to confirm the quality and effectiveness.

show abstract

“…Recently, statistical machine translation (SMT) and Neural Machine Translation (NMT) systems have been the leading machine translation paradigms [ 1 , 2 , 3 ]. Standard SMT techniques do not depend on any linguistic information, and do not apply any pre-processing procedures to generate the translation [ 4 , 5 ].…”

Section: Introductionmentioning

confidence: 99%

Machine Translation Utilizing the Frequent-Item Set Concept

Mahmoud

Mengash

2021

Sensors

View full text Add to dashboard Cite

In this paper, we introduce new concepts in the machine translation paradigm. We treat the corpus as a database of frequent word sets. A translation request triggers association rules joining phrases present in the source language, and phrases present in the target language. It has to be noted that a sequential scan of the corpus for such phrases will increase the response time in an unexpected manner. We introduce the pre-processing of the bilingual corpus through proposing a data structure called Corpus-Trie (CT) that renders a bilingual parallel corpus in a compact data structure representing frequent data items sets. We also present algorithms which utilize the CT to respond to translation requests and explore novel techniques in exhaustive experiments. Experiments were performed on specific language pairs, although the proposed method is not restricted to any specific language. Moreover, the proposed Corpus-Trie can be extended from bilingual corpora to accommodate multi-language corpora. Experiments indicated that the response time of a translation request is logarithmic to the count of unrepeated phrases in the original bilingual corpus (and thus, the Corpus-Trie size). In practical situations, 5–20% of the log of the number of the nodes have to be visited. The experimental results indicate that the BLEU score for the proposed CT system increases with the size of the number of phrases in the CT, for both English-Arabic and English-French translations. The proposed CT system was demonstrated to be better than both Omega-T and Apertium in quality of translation from a corpus size exceeding 1,600,000 phrases for English-Arabic translation, and 300,000 phrases for English-French translation.

show abstract

Character Decomposition for Japanese-Chinese Character-Level Neural Machine Translation

Cited by 6 publications

References 12 publications

Sense-Aware Decoder for Character Based Japanese-Chinese NMT

Sense-Aware Decoder for Character Based Japanese-Chinese NMT

WCC-JC: A Web-Crawled Corpus for Japanese-Chinese Neural Machine Translation

Machine Translation Utilizing the Frequent-Item Set Concept

Contact Info

Product

Resources

About