The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation

Abstract

The past year has witnessed rapid advances in sequence-to-sequence (seq2seq) modeling for Machine Translation (MT). The classic RNN-based approaches to MT were first out-performed by the convolutional seq2seq model, which was then outperformed by the more recent Transformer model. Each of these new approaches consists of a fundamental architecture accompanied by a set of modeling and training techniques that are in principle applicable to other seq2seq architectures. In this paper, we tease apart the new architectures and their accompanying techniques in two ways. First, we identify several key modeling and training techniques, and apply them to the RNN architecture, yielding a new RNMT+ model that outperforms all of the three fundamental architectures on the benchmark WMT'14 English→French and English→German tasks. Second, we analyze the properties of each fundamental seq2seq architecture and devise new hybrid architectures intended to combine their strengths. Our hybrid models obtain further improvements, outperforming the RNMT+ model on both benchmark datasets.

Transformer-Based Acoustic Modeling for Hybrid Speech Recognition

“…There are many works we are yet to explore. For example, our experiments did not show to what extent transformer's superior performance comes from replacing recurrence with self-attention, while other modeling techniques from transformer can be borrowed to improve RNNs as well [42]. The quadratically growing cost with respect to the length of speech signals is still a major blocker for transformer-based acoustic models to be used in practice.…”

Re-Translation Strategies for Long Form, Simultaneous, Spoken Language Translation

“…Our streaming and re-translation models are implemented in Lingvo (Shen et al, 2019), sharing architecture and hyper-parameters wherever possible. Our RNMT+ architecture (Chen et al, 2018) consists of a 6 layer LSTM encoder and an 8 layer Table 2: An example of proportional prefix training. Each example in the minibatch has a 50% chance to be truncated, in which case, we truncate its source and target to a randomly-selected fraction of their original lengths, 1/3 in this example.…”

A Comparative Study on End-to-End Speech to Text Translation

“…We employ frequent casing for the IWSLT tasks while lowercase for the LibriSpeech. There, the evaluation of the IWSLT En→De is case-sensitive, while that of the LibriSpeech is case-insensitive 4 . The translation models are evaluated using the official scripts of WMT campaign, i.e.…”

“…The success of deep neural network (DNN) in both machine translation (MT) [1][2][3][4][5] and automatic speech recognition (ASR) [6][7][8][9][10] has inspired the work of end-to-end speech to text translation (ST) systems. The traditional ST methods are based on a consecutive cascaded pipeline of ASR and MT systems.…”

“…http://www.statmt.org/moses/?n=Moses.SupportTools4 We do the case-insensitive evaluation to be comparable with the other works, however, it is not clear which BLEU script they used[18,23] 5. ftp://jaguar.ncsl.nist.gov/mt/resources/mteval-v13a.pl 6 http://www.cs.umd.edu/ snover/tercom/ 7 http://www1.icsi.berkeley.edu/Speech/docs/sctk-1.2/sclite.htm…”