Uformer: A Unet based dilated complex & real dual-path conformer network for simultaneous speech enhancement and dereverberation

Abstract

Complex spectrum and magnitude are considered as two major features of speech enhancement and dereverberation. Traditional approaches always treat these two features separately, ignoring their underlying relationship. In this paper, we propose Uformer, a Unet based dilated complex & real dual-path conformer network in both complex and magnitude domain for simultaneous speech enhancement and dereverberation. We exploit time attention (TA) and dilated convolution (DC) to leverage local and global contextual information and frequency attention (FA) to model dimensional information. These three sub-modules contained in the proposed dilated complex & real dual-path conformer module effectively improve the speech enhancement and dereverberation performance. Furthermore, hybrid encoder and decoder are adopted to simultaneously model the complex spectrum and magnitude and promote the information interaction between two domains. Encoder decoder attention is also applied to enhance the interaction between encoder and decoder. Our experimental results outperform all SOTA time and complex domain models objectively and subjectively. Specifically, Uformer reaches 3.6032 DNSMOS on the blind test set of Interspeech 2021 DNS Challenge, which outperforms all top-performed models. We also carry out ablation experiments to tease apart all proposed submodules that are most important.

Learning Noise-independent Speech Representation for High-quality Voice Conversion for Noisy Target Speakers

“…Since the clean data of the Realnoisy set is not available, we will not evaluate the topline for the real-world speaker. As for the baseline model, we directly conduct speech enhancement through the state-of-the-art pre-trained Uformer [38] model on all noisy target speakers to obtain denoised speech. For the proposed NC-WaveGAN, we only use the clean and the noisy version of VCTK-clean, where the target noisy speakers in VCTK-noisy and Real-noise are excluded since the WaveGAN model is robust for unseen speakers.…”