Improvements to Deep Convolutional Neural Networks for LVCSR

Abstract

Deep Convolutional Neural Networks (CNNs) are more powerful than Deep Neural Networks (DNN), as they are able to better reduce spectral variation in the input signal. This has also been confirmed experimentally, with CNNs showing improvements in word error rate (WER) between 4-12% relative compared to DNNs across a variety of LVCSR tasks. In this paper, we describe different methods to further improve CNN performance. First, we conduct a deep analysis comparing limited weight sharing and full weight sharing with state-of-the-art features. Second, we apply various pooling strategies that have shown improvements in computer vision to an LVCSR speech task. Third, we introduce a method to effectively incorporate speaker adaptation, namely fMLLR, into log-mel features. Fourth, we introduce an effective strategy to use dropout during Hessian-free sequence training. We find that with these improvements, particularly with fMLLR and dropout, we are able to achieve an additional 2-3% relative improvement in WER on a 50-hour Broadcast News task over our previous best CNN baseline. On a larger 400-hour BN task, we find an additional 4-5% relative improvement over our previous best CNN baseline.

Phone recognition with hierarchical convolutional deep maxout networks

“…Several strategies exist for this, the classic one being maxpooling [7], but other, more sophisticated pooling formulas have also been proposed. For example, Abdel-Hamid et al investigated weighted softmax pooling [11], while Sainath et al studied p-norm pooling and stochastic pooling [12]. However, none of these proved significantly better than simple max-pooling.…”

“…In the full weight sharing scheme (FWS), all neurons are applied across all spectral regions, so the neurons encounter a more elaborate learning task. However, Sainath et al argue that with a large enough number of hidden units, FWS can attain the same performance as LWS, while it is technically simpler and allows the stacking of convolutional layers [12]. In this study, we applied limited weight sharing, which is shown in Fig.…”

“…Extending the shifting to the time axis seems unnecessary, as hidden Markov models inherently handle time shifts. Recently, both Abdel-Hamid et al and Sainath et al experimented with convolution along time, and the improvements indeed proved negligible [11,12].…”

“…Indeed, all the studies that experimented with frequency domain convolution found that CNNs consistently outperform fully connected DNNs on the same task [7][8][9][10]. Later studies experimented with various parameter settings, network structures, and pooling strategies, including time domain convolution [8,[11][12][13]. Also, the experimentation has been extended to large vocabulary recognition (LVCSR) tasks, and the latest results show that CNNs can bring a 12-14 % relative improvement in the word error rate over DNNs trained on the same LVCSR dataset [14].…”

“…First, all the abovementioned studies built the convolutional networks out of sigmoid neurons [7,9] or rectified linear units (ReLUs) [12,14]. However, a novel type of neural activation function called the maxout activation has been recently proposed [15].…”