Discriminatively trained joint speaker and environment representations for adaptation of deep neural network acoustic models

Yin, Maofan; Sivadas, Sunil; Yu, Kai; Ma, Bin

doi:10.1109/icassp.2016.7472642

Cited by 4 publications

(2 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In some complicated network such like convolution-LSTMdeep neural network (CLDNN) [30] [31] and multi-task network [32], different parts of the network can have different beta stabilizers. [13] also mentioned that beta stabilizer can be used not only for SGD but also other training algorithm such like AdaGrad and AdaDelta.…”

Section: Discussionmentioning

confidence: 99%

An investigation on deep learning with beta stabilizer

Liu¹,

Tan²,

Yu³

2016

2016 IEEE 13th International Conference on Signal Processing (ICSP)

Self Cite

View full text Add to dashboard Cite

Artificial neural networks (ANN) have been used in many applications such like handwriting recognition and speech recognition. It is well-known that learning rate is a crucial value in the training procedure for artificial neural networks. It is shown that the initial value of learning rate can confoundedly affect the final result and this value is always set manually in practice. A new parameter called beta stabilizer has been introduced to reduce the sensitivity of the initial learning rate. But this method has only been proposed for deep neural network (DNN) with sigmoid activation function. In this paper we extended beta stabilizer to long short-term memory (LSTM) and investigated the effects of beta stabilizer parameters on different models, including LSTM and DNN with relu activation function. It is concluded that beta stabilizer parameters can reduce the sensitivity of learning rate with almost the same performance on DNN with relu activation function and LSTM. However, it is shown that the effects of beta stabilizer on DNN with relu activation function and LSTM are fewer than the effects on DNN with sigmoid activation function.

show abstract

Section: Discussionmentioning

confidence: 99%

An investigation on deep learning with beta stabilizer

Liu¹,

Tan²,

Yu³

2016

2016 IEEE 13th International Conference on Signal Processing (ICSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…It is observed that these two networks can be trained together. Multi-task training [30,31,32] is a suitable method for joint training.…”

Section: Multi-task Enhanced Lstmmentioning

confidence: 99%

Future vector enhanced LSTM language model for LVCSR

Liu¹,

Ye²,

Yu³

2017

2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)

Self Cite

View full text Add to dashboard Cite

Language models (LM) play an important role in large vocabulary continuous speech recognition (LVCSR). However, traditional language models only predict next single word with given history, while the consecutive predictions on a sequence of words are usually demanded and useful in LVCSR. The mismatch between the single word prediction modeling in trained and the long term sequence prediction in read demands may lead to the performance degradation. In this paper, a novel enhanced long short-term memory (LSTM) LM using the future vector is proposed. In addition to the given history, the rest of the sequence will be also embedded by future vectors. This future vector can be incorporated with the LSTM LM, so it has the ability to model much longer term sequence level information. Experiments show that, the proposed new LSTM LM gets a better result on BLEU scores for long term sequence prediction. For the speech recognition rescoring, although the proposed LSTM LM obtains very slight gains, the new model seems obtain the great complementary with the conventional LSTM LM. Rescoring using both the new and conventional LSTM LMs can achieve a very large improvement on the word error rate.

show abstract

An Investigation on Deep Learning with Beta Stabilizer

Liu

Tan

2020

Preprint

View full text Add to dashboard Cite

Discriminatively trained joint speaker and environment representations for adaptation of deep neural network acoustic models

Cited by 4 publications

References 17 publications

An investigation on deep learning with beta stabilizer

An investigation on deep learning with beta stabilizer

Future vector enhanced LSTM language model for LVCSR

An Investigation on Deep Learning with Beta Stabilizer

Contact Info

Product

Resources

About