Fullsubnet: A Full-Band and Sub-Band Fusion Model for Real-Time Single-Channel Speech Enhancement

Abstract

This paper proposes a full-band and sub-band fusion model, named as FullSubNet, for single-channel real-time speech enhancement. Full-band and sub-band refer to the models that input full-band and sub-band noisy spectral feature, output full-band and sub-band speech target, respectively. The sub-band model processes each frequency independently. Its input consists of one frequency and several context frequencies. The output is the prediction of the clean speech target for the corresponding frequency. These two types of models have distinct characteristics. The full-band model can capture the global spectral context and the long-distance crossband dependencies. However, it lacks the ability to modeling signal stationarity and attending the local spectral pattern. The sub-band model is just the opposite. In our proposed FullSubNet, we connect a pure full-band model and a pure sub-band model sequentially and use practical joint training to integrate these two types of models' advantages. We conducted experiments on the DNS challenge (INTERSPEECH 2020) dataset to evaluate the proposed method. Experimental results show that full-band and sub-band information are complementary, and the FullSubNet can effectively integrate them. Besides, the performance of the FullSubNet also exceeds that of the top-ranked methods in the DNS Challenge (INTERSPEECH 2020).

Learning Deep Direct-Path Relative Transfer Function for Binaural Sound Source Localization

“…In addition, the inter-channel magnitude/intensity difference plays an especially important role for binaural localization, as the intensity difference of binaural signals can reflect the torso/head shadow effect of signal propagation. In order to promote the localization performance, the recently proposed FullSubNet [16] is adopted to predict the complex ideal ratio mask and enhance the complex speech spectrograms. Accounting for the following DP-RTF learning, the clean direct-path sound is taken as the target signal, which means both noise reduction and dereverberation are conducted.…”

“…The enhanced speech would be definitely helpful for DP-RTF estimation. In this work, we adopt the network architecture of the monaural speech enhancement method in [16]. This enhancement method is modified to recover the clean directpath magnitude and phase spectrograms from the contaminated ones, instead of recovering the noise-free signals.…”

YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone

“…As the authors did not use a studio with sound isolation, the dataset contains some environment noise. For our experiments we resample the audios to 16Khz and use the FullSubNet model [34] as denoiser. For development we randomly selected 500 samples and the rest of the dataset was used for training.…”

Improving Character Error Rate Is Not Equal to Having Clean Speech: Speech Enhancement for ASR Systems with Black-box Acoustic Models

“…We utilized the small part in our experiments, specifically, 9h worth (2477 utterances) for training and the remaining 1h (286 utterances) for inference and evaluation. This division between the training and evaluation sets, i.e., into 9h and 1h sets, is given in [24] and its code 3 .…”

“…For instance, convolutional neural networks (CNNs) [1] have been shown to be better than using a short-time Fourier transform (STFT) and inverse STFT (ISTFT) for building an encoder and decoder [2]. Furthermore, methods that utilize recurrent neural networks (RNNs)-based models have been shown to be capable of real-time processing [3][4][5]. In addition, there are hybrid methods that exploit the benefits of both types of network, i.e., real-time processing and high performance [6,7].…”