Parallel Neural Network Features for Improved Tandem Acoustic Modeling

Tüske, Zoltán; Michel, Wilfried; Schlüter, Ralf; Ney, Hermann

doi:10.21437/interspeech.2017-1747

Cited by 8 publications

(7 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our cross validation (CV) set was defined only on the Switchboard part, randomly selecting around 10% of the recordings. The details of the speaker adaptive and discriminatively (MPE-SA) trained acoustic model (AM) are described in [18]. Perplexities (PPL) and recognition results are reported on the complete Hub5 2000 (Hub5'00) test set.…”

Section: Experimental Setup 21 Speech Corporamentioning

confidence: 99%

See 1 more Smart Citation

Investigation on LSTM Recurrent N-gram Language Models for Speech Recognition

2018

Self Cite

View full text Add to dashboard Cite

Recurrent neural networks (NN) with long short-term memory (LSTM) are the current state of the art to model long term dependencies. However, recent studies indicate that NN language models (LM) need only limited length of history to achieve excellent performance. In this paper, we extend the previous investigation on LSTM network based n-gram modeling to the domain of automatic speech recognition (ASR). First, applying recent optimization techniques and up to 6-layer LSTM networks, we improve LM perplexities by nearly 50% relative compared to classic count models on three different domains. Then, we demonstrate by experimental results that perplexities improve significantly only up to 40-grams when limiting the LM history. Nevertheless, the ASR performance saturates already around 20-grams despite across sentence modeling. Analysis indicates that the performance gain of LSTM NNLM over count models results only partially from the longer context and cross sentence modeling capabilities. Using equal context, we show that deep 4-gram LSTM can significantly outperform large interpolated count models by performing the backing off and smoothing significantly better. This observation also underlines the decreasing importance to combine state-of-the-art deep NNLM with count based model.

show abstract

Section: Experimental Setup 21 Speech Corporamentioning

confidence: 99%

“…The task focused on recognizing German Skype conversations. The speaker adaptive and sequence discriminative AMs were trained according to [18]. For this task, we train language models on a corpus of 1-billion words covering 11 different domains [25].…”

Section: Experimental Setup 21 Speech Corporamentioning

confidence: 99%

Investigation on LSTM Recurrent N-gram Language Models for Speech Recognition

2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…The rest of the transcriptions, which amounts to 26.7 M running words, are used as training data for all language models: both the 4-gram Kneser Ney count model (KN4) [23] and neural models. This selection is the same as in [24]. A vocabulary size of 30K is used.…”

Section: Data Descriptionmentioning

confidence: 99%

“…However, the gap disappears after distillation. Our baseline ASR setup is based on the system presented in [24]. From that system, we only use one acoustic model based on a 5-layer bidirectional LSTM-RNN with 500 nodes in each layer and the rescoring pipeline is simplified by applying the lattice rescoring only once using a single neural LM.…”

Section: Mlp Vs Cnn As the Student Modelmentioning

confidence: 99%

Prediction of LSTM-RNN Full Context States as a Subtask for N-Gram Feedforward Language Models

Irie

Lei

Schlüter

et al. 2018

2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Long short-term memory (LSTM) recurrent neural network language models compress the full context of variable lengths into a fixed size vector. In this work, we investigate the task of predicting the LSTM hidden representation of the full context from a truncated n-gram context as a subtask for training an n-gram feedforward language model. Since this approach is a form of knowledge distillation, we compare two methods. First, we investigate the standard transfer based on the Kullback-Leibler divergence of the output distribution of the feedforward model from that of the LSTM. Second, we minimize the mean squared error between the hidden state of the LSTM and that of the n-gram feedforward model. We carry out experiments on different subsets of the Switchboard speech recognition dataset for feedforward models with a short (5-gram) and a medium (10-gram) context length. We show that we get improvements in perplexity and word error rate of up to 8% and 4% relative for the medium model, while the improvements are only marginal for the short model.

show abstract

“…Main experiments are conducted on the 300 hour Switchboard English conversational telephone speech task [12] being the most studied ASR benchmark today [2,3,19,21,[25][26][27]. We used Switchboard-1 Release 2 (LDC97S62) as the training set.…”

Section: Datasetsmentioning

confidence: 99%

An Investigation of Mixup Training Strategies for Acoustic Models in ASR

Medennikov

Khokhlov²,

Romanenko

et al. 2018

Interspeech 2018

View full text Add to dashboard Cite

Mixup is a recently proposed technique that creates virtual training examples by combining existing ones. It has been successfully used in various machine learning tasks. This paper focuses on applying mixup to automatic speech recognition (ASR). More specifically, several strategies for acoustic model training are investigated, including both conventional cross-entropy and novel lattice-free MMI models. Considering mixup as a method of data augmentation as well as regularization, we compare it with widely used speed perturbation and dropout techniques. Experiments on Switchboard-1, AMI and TED-LIUM datasets shows consistent improvement of word error rate up to 13% relative. Moreover, mixup is found to be particularly effective on test data mismatched to the training data.

show abstract

Parallel Neural Network Features for Improved Tandem Acoustic Modeling

Cited by 8 publications

References 44 publications

Investigation on LSTM Recurrent N-gram Language Models for Speech Recognition

Investigation on LSTM Recurrent N-gram Language Models for Speech Recognition

Prediction of LSTM-RNN Full Context States as a Subtask for N-Gram Feedforward Language Models

An Investigation of Mixup Training Strategies for Acoustic Models in ASR

Contact Info

Product

Resources

About