Robust ASR using neural network based speech enhancement and feature simulation

Abstract

We consider the problem of robust automatic speech recognition (ASR) in the context of the CHiME-3 Challenge. The proposed system combines three contributions. First, we propose a deep neural network (DNN) based multichannel speech enhancement technique, where the speech and noise spectra are estimated using a DNN based regressor and the spatial parameters are derived in an expectation-maximization (EM) like fashion. Second, a conditional restricted Boltzmann machine (CRBM) model is trained using the obtained enhanced speech and used to generate simulated training and development datasets. The goal is to increase the similarity between simulated and real data, so as to increase the benefit of multicondition training. Finally, we make some changes to the ASR backend. Our system ranked 4th among 25 entries.

An analysis of environment, microphone and data simulation mismatches in robust speech recognition

“…More work on ground truth estimation is required to close this gap and benefit from simulated training data. In addition, training on real data now leads to a performance decrease on simulated data, while Sivasankaran et al (2015) found it to consistently improve performance on both real and simulated data. Along with the recent results of Nugraha et al (2016b) on another dataset, this suggests that, although weighted EM made little difference for spectral models other than DNN (Liutkus et al, 2015), weighted EM outperforms exact EM for the estimation of multichannel statistics from DNN outputs.…”

“…One possible explanation for the difference observed when training the enhancement DNN of Sivasankaran et al (2015) on real vs. simulated data may be the way the ground truth is estimated rather than the data themselves. Indeed, as shown in Section 2.3.3, the spectrograms of real and simulated data appear to be similar, while the underlying ground truth speech signals, which are estimated from noisy and close-talk signals in the case of real data, look quite different.…”

“…In the following, we do not discuss DNN post-filters, which provided a limited improvement or degradation on both real and simulated data (Hori et al, 2015;Sivasankaran et al, 2015), and we focus on multichannel DNN-based enhancement instead. Table 5 illustrates the performance of the DNN-based time-invariant generalized eigenvalue (GEV) beamformer proposed by Heymann et al (2015).…”

“…These results also indicate that the enhancement system is able to leverage the simulated data to learn about the real data and that increasing the amount and variety of simulated data further improves performance. Sivasankaran et al (2015) exploited a DNN to perform multichannel timevarying Wiener filtering instead. The desired DNN outputs are the magnitude spectra of speech and noise, which are computed from the underlying clean speech signals in the case of simulated data or using the procedure described in Section 2.3.1 in the case of real data.…”

“…We performed this experiment using the DNN-based multichannel source separation technique of Nugraha et al (2016a), which is a variant of the one of Sivasankaran et al (2015) that relies on exact EM updates for the spatial covariance matrices (Duong et al, 2010) instead of the weighted EM updates of Liutkus et al (2015).…”