The NTT CHiME-3 system: Advances in speech enhancement and recognition for mobile multi-microphone devices

An analysis of environment, microphone and data simulation mismatches in robust speech recognition

“…Table 5 illustrates the performance of the DNN-based time-invariant generalized eigenvalue (GEV) beamformer proposed by Heymann et al (2015). This beamformer is similar to the mask-based MVDR beamformer of Yoshioka et al (2015) mentioned in Section 3.2.1, except that the time-frequency mask from which the multichannel statistics of speech and noise are computed is estimated via a DNN instead of a clustering technique. It is followed by a time-invariant blind analytic normalization (BAN) filter which rescales the beamformer output to ensure unit gain for the speaker signal.…”

“…A few challenge entries also employed multichannel dereverberation techniques based on time-domain linear prediction (Yoshioka et al, 2010) or interchannel coherence-based time-frequency masking (Schwarz and Kellermann, 2014). As expected, these techniques improved performance on real data but made a smaller difference or even degraded performance on simulated data due to the fact that it did not include any early reflection or reverberation (Yoshioka et al, 2015;Barfuss et al, 2015;Pang and Zhu, 2015).…”

“…Indeed, early reflections (and to a lesser extent reverberation) modify the apparent speaker direction at each frequency, which results Table 4: WER (%) achieved by beamforming and spatial post-filtering applied on all channels except ch2 using the GMM backend retrained on enhanced real and simulated data (Prudnikov et al, 2015 (Anguera et al, 2007) which was used in many challenge submissions do not suffer from this issue due to the fact that their spatial response decays slowly in the neighborhood of the estimated speaker direction. Modern adaptive beamformers such as MCA or the mask-based MVDR beamformer of Yoshioka et al (2015) do not suffer from this issue either, due to the fact that they estimate the relative (inter-microphone) transfer function instead of the direction-of-arrival. Specifically, Yoshioka et al (2015) estimated a time-frequency mask which represents the proposition of speech vs. noise in every time-frequency bin and they derived the beamformer from the multichannel statistics (spatial covariance matrices) of speech and noise computed from the corresponding time-frequency bins.…”

“…Modern adaptive beamformers such as MCA or the mask-based MVDR beamformer of Yoshioka et al (2015) do not suffer from this issue either, due to the fact that they estimate the relative (inter-microphone) transfer function instead of the direction-of-arrival. Specifically, Yoshioka et al (2015) estimated a time-frequency mask which represents the proposition of speech vs. noise in every time-frequency bin and they derived the beamformer from the multichannel statistics (spatial covariance matrices) of speech and noise computed from the corresponding time-frequency bins. They reported this beamformer to perform similarly on real and simulated data, which is particularly noticeable as it contributed to their entry winning the challenge.…”

“…The performance improvements brought by these increasingly complex architectures appear to be consistent on real and simulated data. Table 8: WER (%) achieved on noisy data using various acoustic models trained on noisy real and simulated data (all channels) without sMBR (Yoshioka et al, 2015 It must be noted that, with the exception of Vu et al (2015), all challenge entrants trained GMM-HMM and DNN-HMM acoustic models on real and simulated data. Heymann et al (2015) found that discarding real data and training a GMM-HMM acoustic model on simulated data only increases the WER by 3% and 4% relative on real development and test data, respectively.…”