RETURNN as a Generic Flexible Neural Toolkit with Application to Translation and Speech Recognition

Abstract

We compare the fast training and decoding speed of RETURNN of attention models for translation, due to fast CUDA LSTM kernels, and a fast pure Tensor-Flow beam search decoder. We show that a layer-wise pretraining scheme for recurrent attention models gives over 1% BLEU improvement absolute and it allows to train deeper recurrent encoder networks. Promising preliminary results on max. expected BLEU training are presented. We obtain state-of-the-art models trained on the WMT 2017 German↔English translation task. We also present end-to-end model results for speech recognition on the Switchboard task. The flexibility of RETURNN allows a fast research feedback loop to experiment with alternative architectures, and its generality allows to use it on a wide range of applications.

Improved Multi-Stage Training of Online Attention-Based Encoder-Decoder Models

“…An MP layer has no trainable parameters, but helps primarily in reducing the length of an input sequence by a predetermined pooling factor. It has been observed that reducing the input sequence length progressively through the encoder helps in better convergence and accuracy of the AED models [2,8,16]. In this work, we use an overall time reduction factor of 2 for character encoder, and an additional reduction factor of 4 in the BPE stack, taking the overall reduction factor for the BPE encoder to 8.…”

“…In [9], a teacher-student transfer learning has been used to improve the convergence as well as the performance of character based CTC encoder models. In the case of AED models, it has been shown that a carefully designed layerwise pre-training strategy helps the models to converge better [2]. Similarly, a joint multi-task training using a CTC loss on the encoder and a CE loss on the attention decoder was proposed to improve the convergence and accuracies of AED models [19,20].…”

“…Our decoder has a single ULSTM layer. We also use attention feedback and a maxout layer similar to to [2]. This whole network is now trained with the cross-entropy loss between decoder output and the ground truth BPE target labels.…”

“…Our decoder consists of a single ULSTM layer with 1024 units. We use RETURNN framework for all our experiments [26,2,27]. In all our experiments we use Adam optimizer [28] with an initial learning rate of 0.0008 and learning rate scheduling using a cross-validation set.…”

“…Recently, attention-based encoder-decoder (AED) models have gained popularity for developing end-to-end neural network based automatic speech recognition (ASR) systems [1,2,3]. One of the primary advantages of AED models is that the language information is tightly coupled into the decoder, obviating the need for an external language model (LM).…”