Output-Gate Projected Gated Recurrent Unit for Speech Recognition

Cheng, Gaofeng; Povey, Daniel; Huang, Lü; Ji, Xu; Khudanpur, Sanjeev; Yan, Yonghong

doi:10.21437/interspeech.2018-1403

Cited by 17 publications

(12 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The details of baseline models in the experiments are shown in Table 4. The network architecture of Time Delay Neural Network-Long Short-Term Memory with Projection(TDNN-LSTMP), Bidirectional Long Short-Term Memory with Projection (BLSTMP), and Time Delay Neural Network-Output gate Projected Gated Recurrent Unit(TDNN-OPGRU) are described in [20] [21] [22], respectively. As [24] does not give the details of the Time Delay Neural Networkminimal Gated Recurrent Unit (TDNN-mGRU), when we set up the architecture of TDNN-mGRU, the parameters of TDNN are set as following the context module parameters of mGRUIP-Ctx [32], and the cell dimension is consistent with the setting of LSTMP in TDNN-LSTMP [20].…”

Section: Comparison Between Norm-pmgru and Baseline Modelsmentioning

confidence: 99%

“…Thus, considering the ability to model a dynamic window of all sequence history instead of a fixed contextual window over the input sequence as FFNN does, RNN is more suitable for sequence modeling. However, as problems of vanishing gradient and exploding gradient highly happen in the vanilla RNN training process, Long Short-Term Memory (LSTM) [18], Gated Recurrent Unit (GRU) [19] and their Long Short-Term Memory with Projection (LSTMP) [20], Bidirectional Long Short-Term Memory with Projection (BLSTMP) [21], Output-Gate Projected Gated Recurrent Unit (OPGRU) [22], minimal Gated Recurrent Unit (mGRU, also named LiGRU) [23] [24], mGRUIP with Context module (mGRUIP-Ctx) [25], are proposed to solve the above problems and well used in speech recognition field. In [21], the author compared the performance of TDNN-LSTMP and BLSTMP.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Projected Minimal Gated Recurrent Unit for Speech Recognition

Ran

Jiang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Recurrent neural network (RNN) has the ability to learn long-term dependencies, which makes it suitable for acoustic modeling in speech recognition. In this paper, we revise RNN model used in acoustic modeling, namely, mGRUIP with Context module (mGRUIP-Ctx), and propose an advanced model which named Projected minimal Gated Recurrent Unit (PmGRU). The paper demonstrates two major contributions: firstly, in the case that adding context information to context module in mGRUIP-Ctx will bring about large amount of parameter, we propose to insert a smaller output projection layer after the mGRUIP-Ctx cell's output to form the PmGRU, which is inspired by the idea of low-rank decomposition of matrix. The output projection layer has been proved to be able to save most of the effective information with the reduction of model parameters. Secondly, in the case that too much context information of previous layer introduced by context module will cause declining of model performance, we adjust the ratio of context information of the previous layer to the current layer by moving the position of batch normalization layer, and the final RNN model Normalization Projected minimal Gated Recurrent Unit (Norm-PmGRU) is generated. In the five automatic speech recognition (ASR) tasks, the Norm-PmGRU has been proved more effectively in the experiments compared with mGRUIP-Ctx, TDNN-OPGRU, TDNN-LSTMP and other RNN baseline acoustics models. INDEX TERMS Speech recognition, acoustic model, gated recurrent unit, batch normalization.

show abstract

Section: Comparison Between Norm-pmgru and Baseline Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Projected Minimal Gated Recurrent Unit for Speech Recognition

Ran

Jiang

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…4. Various network architectures were used through combinations of different types of layers, including TDNN, LSTM, Output-gate Projected Gated Recurrent Unit(BOPGRU) [17], CNN, TDNNF [18], Residual Bidirectional LSTM(RBiLSTM) [19] and other techniques, such as self-attention mechanism [20] and backstitch [21]. We evaluated different model architectures on the development set.…”

Section: Am Constructionmentioning

confidence: 99%

“…CNN-TDNN-RBiLSTM( Fig.2c): the architecture was proposed in [17] which backward (b)-LSTM was applied on top of the forward(f)-LSTM and directly appending the outputs of f-LSTM and b-LSTM (Fig.2b).…”

Section: Am Constructionmentioning

confidence: 99%

The LeVoice Far-Field Speech Recognition System for VOiCES from a Distance Challenge 2019

Liang¹,

Yang²,

Wang³

et al. 2019

Interspeech 2019

View full text Add to dashboard Cite

This paper describes our submission to the "VOiCES from a Distance Challenge 2019", which is designed to foster research in the area of speaker recognition and automatic speech recognition (ASR) with a special focus on single channel distant/far-field audio under noisy conditions. We focused on the ASR task under a fixed condition in which the training data was clean and small, but the development data and test data were noisy and unmatched. Thus we developed the following major technical points for our system, which included data augmentation, weighted-prediction-error based speech enhancement, acoustic models based on different networks, TDNN or LSTM based language model rescore, and ROVER. Experiments on the development set and the evaluation set showed that the front-end processing, data augmentation and system fusion made the main contributions for the performance increasing, and the final word error rate results based on our system scored 15.91% and 19.6% respectively.

show abstract

“…LF-MMI and LF-bMMI were adopted in [13] and [14] respectively on English conversation transcription and multi-talker speech recognition task but in both work cross-entropy pre-training is still required. Later on, various model architectures [15,16,17] for LF-MMI have been explored. LF-sMBR was proposed and compared against lattice-based sMBR in [18] when initialized from a LF-MMI trained model.…”

Section: Introductionmentioning

confidence: 99%

A Comparison of Lattice-free Discriminative Training Criteria for Purely Sequence-trained Neural Network Acoustic Models

Weng

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

In this work, three lattice-free (LF) discriminative training criteria for purely sequence-trained neural network acoustic models are compared on LVCSR tasks, namely maximum mutual information (MMI), boosted maximum mutual information (bMMI) and state-level minimum Bayes risk (sMBR). We demonstrate that, analogous to LF-MMI, a neural network acoustic model can also be trained from scratch using LF-bMMI or LF-sMBR criteria respectively without the need of cross-entropy pre-training. Furthermore, experimental results on Switchboard-300hrs and Switchboard+Fisher-2100hrs datasets show that models trained with LF-bMMI consistently outperform those trained with plain LF-MMI and achieve a relative word error rate (WER) reduction of ∼5% over competitive temporal convolution projected LSTM (TDNN-LSTMP) LF-MMI baselines.

show abstract

Output-Gate Projected Gated Recurrent Unit for Speech Recognition

Cited by 17 publications

References 17 publications

Projected Minimal Gated Recurrent Unit for Speech Recognition

Projected Minimal Gated Recurrent Unit for Speech Recognition

The LeVoice Far-Field Speech Recognition System for VOiCES from a Distance Challenge 2019

A Comparison of Lattice-free Discriminative Training Criteria for Purely Sequence-trained Neural Network Acoustic Models

Contact Info

Product

Resources

About