Light Gated Recurrent Units for Speech Recognition

Abstract

A field that has directly benefited from the recent advances in deep learning is Automatic Speech Recognition (ASR). Despite the great achievements of the past decades, however, a natural and robust human-machine speech interaction still appears to be out of reach, especially in challenging environments characterized by significant noise and reverberation. To improve robustness, modern speech recognizers often employ acoustic models based on Recurrent Neural Networks (RNNs), that are naturally able to exploit large time contexts and long-term speech modulations. It is thus of great interest to continue the study of proper techniques for improving the effectiveness of RNNs in processing speech signals.In this paper, we revise one of the most popular RNN models, namely Gated Recurrent Units (GRUs), and propose a simplified architecture that turned out to be very effective for ASR. The contribution of this work is two-fold: First, we analyze the role played by the reset gate, showing that a significant redundancy with the update gate occurs. As a result, we propose to remove the former from the GRU design, leading to a more efficient and compact single-gate model. Second, we propose to replace hyperbolic tangent with ReLU activations. This variation couples well with batch normalization and could help the model learn long-term dependencies without numerical issues.Results show that the proposed architecture, called Light GRU (Li-GRU), not only reduces the per-epoch training time by more than 30% over a standard GRU, but also consistently improves the recognition accuracy across different tasks, input features, noisy conditions, as well as across different ASR paradigms, ranging from standard DNN-HMM speech recognizers to endto-end CTC models.

SiTGRU: Single-Tunnelled Gated Recurrent Unit for Abnormality Detection

“…It allows the network to either forward the information from previous state or completely leave if deemed unnecessary. The reset gate is ideal for scenarios where considerable interruptions are manifested in sequential data [30]. For such cases, the reset gate can be turned on fully to attain values closer to zero so as to diminish the impact of the previous memory state on computation of candidate memory state.…”

“…Ravanelli et al [30] assessed the functionality of update and reset gates for speech recognition process, and argue that redundancies might occur in the activations of these two gating mechanisms. Setting similar large or small values to both update and reset gates produces similar effect on candidate memory and current memory states.…”

“…Standard GRUs use hyperbolic tangent function for activation, but these functions are shown to be ineffective for training feed-forward connections as they suffer from performance drop when the network gets deeper [15]. Similarly, Rectified Linear Unit (ReLU) activations portray numerical instabilities emanating from their unbounded nature when used for long range sequences [30]. We thus employ sigmoid activation to compute candidate memory state.…”

“…Recently, Recurrent Neural Networks (RNNs) have gaining popularity due to their ability of handling sequential data. Generalized RNN and its variant Long Short-Term Memory (LSTM) are widely used for AED and other time sequence tasks such as speech recognition [8,30]. In addition, over the past years standard RNNs and LSTMs have gone through several structural modifications which led to the rise of a simplified version of LSTM, called GRU.…”

“…In addition, over the past years standard RNNs and LSTMs have gone through several structural modifications which led to the rise of a simplified version of LSTM, called GRU. GRUs are widely applied and have shown their effectiveness for speech processing, medical image processing and video segmentation tasks [14,30,32]. Despite their consumption for different computer vision problems, abnormal event detection has not taken full advantage of GRUs' inherent nature to develop optimized and effective models, and much effort has not been put forth to exploit GRUs for this task.…”