Speech Enhancement with LSTM Recurrent Neural Networks and its Application to Noise-Robust ASR

Abstract

Abstract. We evaluate some recent developments in recurrent neural network (RNN) based speech enhancement in the light of noise-robust automatic speech recognition (ASR). The proposed framework is based on Long Short-Term Memory (LSTM) RNNs which are discriminatively trained according to an optimal speech reconstruction objective. We demonstrate that LSTM speech enhancement, even when used 'naïvely' as front-end processing, delivers competitive results on the CHiME-2 speech recognition task. Furthermore, simple, feature-level fusion based extensions to the framework are proposed to improve the integration with the ASR back-end. These yield a best result of 13.76 % average word error rate, which is, to our knowledge, the best score to date.

An analysis of environment, microphone and data simulation mismatches in robust speech recognition

“…For traditional speech enhancement techniques, which require either no training or training on the noise context preceding each test utterance (Cohen et al, 2010;Hurmalainen et al, 2013), the issue of mismatched noise conditions did not arise. This recently became a concern with the emergence of speech enhancement techniques based on deep neural networks (DNNs) Xu et al, 2014;Weninger et al, 2015), which require a larger amount of training data not limited to the immediate context. Chen et al (2015) and Kim and Smaragdis (2015) considered the problem of adapting DNN based enhancement to unseen test conditions, but their experiments were conducted on small, simulated datasets and evaluated in terms of enhancement metrics.…”

Feature mapping using far-field microphones for distant speech recognition

“…In [40], the DNN is used to estimate the instantaneous SNR for computing IRM, subsequently applied to filter out noise from a noisy Mel spectrogram. The recurrent neural networks (RNNs), with their ability to model the temporal dependencies in speech, have also been employed to estimate the time-frequency masks from the magnitude spectrum of a noisy signal for speech enhancement and recognition [41]. For speech recognition applications, the speech enhancement approaches are typically exploited as front-end processing to reconstruct the clean version of the speech, which is then fed into a speech recognizer.…”

A perceptually-weighted deep neural network for monaural speech enhancement in various background noise conditions

“…speech recognition [2] and speech synthesis [3]. More recently, the DNN has been applied to speech separation [4]- [7] and enhancement/denoising [8]- [10], particularly for monaural recordings [4]- [6], [8]- [10]. When processing mixtures of target speech signals and competing noise, speech separation may be considered as speech enhancement.…”

“…In order to recover the underlying target speech embedded in noise, most of the deep neural networks, either recurrent [4], [5], [10] or feedforward [4], [6], [8], [9], [11], are trained to optimize some objective functions such as the mean squared error (MSE) between the true and predicted outputs. The inputs to the DNN are often (hybrid) features such as timefrequency (TF) domain spectral features [4]- [6], [8]- [10] and filterbank features [4], [5], [11]; while the output can be the TF unit level features that can be used to recover the speech source, such as ideal binary/ratio masks (IBM/IRM) [4]- [6], [11], direct magnitude spectra [9], [10] or their transforms such as log power (LP) spectra [8].…”

“…The inputs to the DNN are often (hybrid) features such as timefrequency (TF) domain spectral features [4]- [6], [8]- [10] and filterbank features [4], [5], [11]; while the output can be the TF unit level features that can be used to recover the speech source, such as ideal binary/ratio masks (IBM/IRM) [4]- [6], [11], direct magnitude spectra [9], [10] or their transforms such as log power (LP) spectra [8].…”