A Simultaneous Denoising and Dereverberation Framework with Target Decoupling

Abstract

Background noise and room reverberation are regarded as two major factors to degrade the subjective speech quality. In this paper, we propose an integrated framework to address simultaneous denoising and dereverberation under complicated scenario environments. It adopts a chain optimization strategy and designs four sub-stages accordingly. In the first two stages, we decouple the multi-task learning w.r.t. complex spectrum into magnitude and phase, and only implement noise and reverberation removal in the magnitude domain. Based on the estimated priors above, we further polish the spectrum in the third stage, where both magnitude and phase information are explicitly repaired with the residual learning. Due to the data mismatch and nonlinear effect of DNNs, the residual noise often exists in the DNN-processed spectrum. To resolve the problem, we adopt a light-weight algorithm as the post-processing module to capture and suppress the residual noise in the non-active regions. In the Interspeech 2021 Deep Noise Suppression (DNS) Challenge, our submitted system ranked top-1 for the real-time track in terms of Mean Opinion Score (MOS) with ITU-T P.835 framework.

Uformer: A Unet based dilated complex & real dual-path conformer network for simultaneous speech enhancement and dereverberation

“…Four evaluation metrics are used, namely perceptual evaluation of speech quality (PESQ), extended short-time objective intelligibility (eSTOI), DNSMOS [32] [18], namely denoising, dereverberation, spectral refinement and post-processing, respectively. Uformer gives the best performance in both objective and subjective evaluation, while the result of the causal version doesn't degrade generally.…”

“…We conduct ablation experiments to prove the effectiveness of each proposed sub-modules, including a) Uformer without FA, b) Uformer without DC, c) Uformer without encoder decoder attention, d) substitute dilated complex & real dual-path conformer with complex & real LSTM, e) Uformer without all real-valued sub-modules (means we only model complex spectrum) and f) Uformer without all complex-valued sub-modules (means we only model magnitude). We also compare Uformer with DCCRN, PHASEN [28], GCRN, SDD-Net [18], TasNet and DPRNN [29]. DCCRN, PHASEN and GCRN are in complex domain, which aim to model magnitude and phase simultaneously.…”

“…Our experimental results outperform all SOTA time domain and complex domain speech front-end models on both objective and subjective performance. Specifically, Uformer reaches 3.6032 DNS-MOS on the blind test set of Interspeech 2021 DNS challenge, which surpass SDD-Net [18], the top-performed model in the challenge. We also carry out ablation study to tease apart all proposed sub-modules that are most important.…”

Joint magnitude estimation and phase recovery using Cycle-in-Cycle GAN for non-parallel speech enhancement

“…The 512-point STFT is utilized, leading to a 257-D spectral feature. Due to the efficacy of power compression in dereverberation and denoising tasks [21], we conduct the power compression toward the spectral magnitude while leaving the phase unaltered, and the optimal compression coefficient is set to 0.5. For the non-parallel training strategy, we randomly crop a fixed-length segment (i.e., 108 frames) from a randomly selected noisy audio file as the input, while the target is a randomly selected clean audio file whose content is different with the input audio.…”

“…Multi-stage algorithms based on curriculum learning begin to thrive in the SE area [20,21], where the original difficult task is decomposed into multiple simpler sub-tasks and a better result can be induced progressively. Motivated by these studies, we couple a magnitude spectrum estimation CycleGAN (MCGAN) and a complex spectrum refined CycleGAN (CCGAN) as a Cycle-in-Cycle GAN (CinCGAN) paradigm to estimate clean spectral magnitude and phase information step-by-step under non-parallel training.…”