Understanding Pure Character-Based Neural Machine Translation: The Case of Translating Finnish into English

Abstract

Recent work has shown that deeper character-based neural machine translation (NMT) models can outperform subword-based models. However, it is still unclear what makes deeper character-based models successful. In this paper, we conduct an investigation into pure character-based models in the case of translating Finnish into English, including exploring the ability to learn word senses and morphological inflections and the attention mechanism. We demonstrate that word-level information is distributed over the entire character sequence rather than over a single character, and characters at different positions play different roles in learning linguistic knowledge. In addition, character-based models need more layers to encode word senses which explains why only deeper models outperform subword-based models. The attention distribution pattern shows that separators attract a lot of attention and we explore a sparse word-level attention to enforce character hidden states to capture the full word-level information. Experimental results show that the word-level attention with a single head results in 1.2 BLEU points drop.

Interactive machine translation for the language modernization and spelling normalization of historical documents

“…Domingo and Casacuberta [11] evaluated word-based and character-based MT approaches, finding character-based to be more suitable for this task and that SMT systems outperformed NMT systems. Tang et al [53], however, compared many neural architectures and reported that the NMT models are much better than SMT models in terms of character error rate (CER). Finally, Hämäläinen et al [22] evaluated SMT, NMT, an edit-distance approach, and a rule-based finite state transducer and advocated for a combination of these approaches to make use of their individual strengths.…”

Interpretation of Black Box NLP Models: A Survey

“…It provides an insight into the fact that how well a model has learned the specified linguistic property. Several linguistic features have been analysed to extract different properties like Morphological, syntactic and semantic (Voita and Titov, 2020;Tang et al, 2021). It is based on the premise that if there is more task relevant information is learned by the model then it is going to perform better on the presented task.…”

ICT’s System for AutoSimTrans 2021: Robust Char-Level Simultaneous Translation

“…Character-level neural machine translation (Ling et al, 2015;Lee et al, 2017;Cherry et al, 2018;Gao et al, 2020) tokenizes the source sentence and target sentence according to characters, thereby gaining advantages over subword-level neural machine translation in some specific aspects, such as avoiding out-of-vocabulary problems (Passban et al, 2018) and errors caused by subword-level segmentation (Tang et al, 2020). In terms of translation quality, the character-level MT is still difficult to compare with the subword-level MT.…”