Revisiting Character-Based Neural Machine Translation with Capacity and Compression

Abstract

Translating characters instead of words or word-fragments has the potential to simplify the processing pipeline for neural machine translation (NMT), and improve results by eliminating hyper-parameters and manual feature engineering. However, it results in longer sequences in which each symbol contains less information, creating both modeling and computational challenges. In this paper, we show that the modeling problem can be solved by standard sequence-to-sequence architectures of sufficient depth, and that deep models operating at the character level outperform identical models operating over word fragments. This result implies that alternative architectures for handling character input are better viewed as methods for reducing computation time than as improved ways of modeling longer sequences. From this perspective, we evaluate several techniques for characterlevel NMT, verify that they do not match the performance of our deep character baseline model, and evaluate the performance versus computation time tradeoffs they offer. Within this framework, we also perform the first evaluation for NMT of conditional computation over time, in which the model learns which timesteps can be skipped, rather than having them be dictated by a fixed schedule specified before training begins.

On the Importance of Word Boundaries in Character-level Neural Machine Translation

“…Moreover, they are typically deployed as a pre-processing step before training the NMT model, hence, the predicted set of subword units are essentially not optimized for the translation task. Recently, (Cherry et al, 2018) extended the approach of NMT based on subword units to implement the translation model directly at the level of characters, which could reach comparable performance to the subword-based model, although this would require much larger networks which may be more difficult to train. The major reason to this requirement may lie behind the fact that treating the characters as individual tokens at the same level and processing the input sequences in linear time increases the difficulty of the learning task, where translation would then be modeled as a mapping between the characters in two languages.…”

Towards easy-to-implement misinformation automatic detection for online social media

“…As discussed above, we believe its performance is due to its ability to cope with morphologies. Networks using character CNNs have proven a robust alternative when dealing with morphologies in different tasks (Cao and Rei, 2016;Cherry et al, 2018), and we believe this ability is of particular importance for the task at hand. Our analysis showed that networks considering morphologies were better able to categorise misinformation.…”

“…This result is particularly remarkable when considering that networks (1) and (2) have around 60x more parameters than the character CNN. Cao and Rei (2016); Cherry et al (2018) suggest that character CNNs are better equipped than the other networks to cope with morphologies. We put forward this characteristic would be useful when dealing with the particularities of misinformation in online social media.…”

Exploring Phoneme-Level Speech Representations for End-to-End Speech Translation

“…The way translation input is represented has been shown to impact performance as well as how much data the model requires to train (Sennrich et al, 2016;Salesky et al, 2018;Cherry et al, 2018). The current standard approach for textbased translation is to segment words into subword units as a preprocessing step (Sennrich et al, 2016).…”