CMGAN: Conformer-Based Metric-GAN for Monaural Speech Enhancement

Abstract

<p>Convolution-augmented transformers (Conformers) are recently proposed in various speech-domain applications, such as automatic speech recognition (ASR) and speech separation, as they can capture both local and global dependencies. In this paper, we propose a conformer-based metric generative adversarial network (CMGAN) for speech enhancement (SE) in the time-frequency (TF) domain. The generator encodes the magnitude and complex spectrogram information using two-stage conformer blocks to model both time and frequency dependencies. The decoder then decouples the estimation into a magnitude mask decoder branch to filter out unwanted distortions and a complex refinement branch to further improve the magnitude estimation and implicitly enhance the phase information. Additionally, we include a metric discriminator to alleviate metric mismatch by optimizing the generator with respect to a corresponding evaluation score. Objective and subjective evaluations illustrate that CMGAN is able to show superior performance compared to state-of-the-art methods in three speech enhancement tasks (denoising, dereverberation and super-resolution). For instance, quantitative denoising analysis on Voice Bank+DEMAND dataset indicates that CMGAN outperforms various previous models with a margin, i.e., PESQ of 3.41 and SSNR of 11.10 dB. </p>

Multi-Attention Bottleneck for Gated Convolutional Encoder-Decoder-Based Speech Enhancement

“…In addition, they retain complex spectral structures in the final speech. Feedforward DNNs (FDNNs) [5]- [9], [13], [16], [17], CNNs [18], [19], RNNs [20], [21], Generative Adversarial Network (GANs) [22], [23], and Transformers [24]- [26] are successful DNN approaches for speech enhancement.…”

Review of Speech Enhancement Methods using Generative Adversarial Networks

“…CMGAN: Conformer-Based Metric-GAN [23] is a frame-work that operates on both magnitude spectrogram and complex spectrogram components. In addition to a metric discriminator that corrects metric mismatch, it employs conformers that can capture local features along with long-term relationships in both the time and frequency dimensions.…”