Neural Architecture Search for LF-MMI Trained Time Delay Neural Networks

Abstract

Deep neural networks (DNNs) based automatic speech recognition (ASR) systems are often designed using expert knowledge and empirical evaluation. In this paper, a range of neural architecture search (NAS) techniques are used to automatically learn two types of hyperparameters of state-of-the-art factored time delay neural networks (TDNNs): i) the left and right splicing context offsets; and ii) the dimensionality of the bottleneck linear projection at each hidden layer. These include the DARTS method integrating architecture selection with lattice-free MMI (LF-MMI) TDNN training; Gumbel-Softmax and pipelined DARTS reducing the confusion over candidate architectures and improving the generalization of architecture selection; and Penalized DARTS incorporating resource constraints to adjust the trade-off between performance and system complexity. Parameter sharing among candidate architectures allows efficient search over up to 7 28 different TDNN systems. Experiments conducted on the 300-hour Switchboard corpus suggest the auto-configured systems consistently outperform the baseline LF-MMI TDNN systems using manual network design or random architecture search after LHUC speaker adaptation and RNNLM rescoring. Absolute word error rate (WER) reductions up to 1.0% and relative model size reduction of 28% were obtained. Consistent performance improvements were also obtained on a UASpeech disordered speech recognition task using the proposed NAS approaches.

Neural Architecture Search for Speech Emotion Recognition

“…The random search, which selects the best performed architecture from 5 randomly sampled architectures, turns out to be a considerably strong baseline that achieves comparable performance with the system found by joint optimization. This coincides with the observations in previous literature [12,17]. We need to also note that the baseline architectures are already optimized by the authors, which also provides a sufficiently good starting point for the random search.…”

“…Jointly training the architecture weights and the model parameters saves time. However, the sub-optimal operations (e.g., simpler operations) may dominate the weights at an early stage, such that the optimal operations (e.g., operations with larger parameter sizes) may be ignored [17].…”

“…Alternatively, the uniform path sampling strategy is adopted for Eq. ( 4) by [16,17]. The idea is to randomly select one path in the supernet (i.e.…”

“…As the candidate operation parameters and the architecture weights are optimized on the same supernet, the sub-optimal architectures may be prematurely learned at an early stage of the training. To encourage all candidate architectures to be optimized simultaneously, various techniques are applied, e.g., operation dropout [15], uniform path sampling [16,17]. However, the training using operation dropout is often not stable, since certain nodes in the supernet may drop all operations.…”

“…The NAS techniques have also been successfully applied to speech synthesis [18] and speech recognition [17,[19][20][21][22][23][24][25]. It has been shown that the performance can be improved and the model parameter sizes can be reduced at the same time.…”