Differentiable Consistency Constraints for Improved Deep Speech Enhancement

Abstract

In recent years, deep networks have led to dramatic improvements in speech enhancement by framing it as a data-driven pattern recognition problem. In many modern enhancement systems, large amounts of data are used to train a deep network to estimate masks for complex-valued short-time Fourier transforms (STFTs) to suppress noise and preserve speech. However, current masking approaches often neglect two important constraints: STFT consistency and mixture consistency. Without STFT consistency, the system's output is not necessarily the STFT of a time-domain signal, and without mixture consistency, the sum of the estimated sources does not necessarily equal the input mixture. Furthermore, the only previous approaches that apply mixture consistency use real-valued masks; mixture consistency has been ignored for complex-valued masks.In this paper, we show that STFT consistency and mixture consistency can be jointly imposed by adding simple differentiable projection layers to the enhancement network. These layers are compatible with real or complex-valued masks. Using both of these constraints with complex-valued masks provides a 0.7 dB increase in scale-invariant signal-to-distortion ratio (SI-SDR) on a large dataset of speech corrupted by a wide variety of nonstationary noise across a range of input SNRs.

Multi-Microphone Complex Spectral Mapping for Speech Dereverberation

“…3) The motivation is that using ℒ %,'( alone produces worse magnitude estimates, as the estimated magnitudes need to compensate for the estimation error of phase. A major difference from [31], [33] is that we do not perform power or logarithmic compression on the magnitude spectra. This way, the DNN is always trained to estimate an STFT spectrogram that has consistent phase and magnitude structure, and hence would likely produce a good consistent STFT spectrogram at run time [34], [35].…”

Improving Universal Sound Separation Using Sound Classification

“…In the iterative version (iTDCN++), the process of estimating the source signals is repeated twice. The mixture audio m and output estimatesŝ (1) of the first TDCN++ module serve as input for the second TDCN++ module, which produces the final separation estimatesŝ (2) . Both separation networks are trained using permutation-invariant [5] negative signal to noise ratio (SNR), given by…”

“…This extends the aforementioned end-to-end approach in order to predict estimates of the clean sourcesŝ (1) . These estimates, as well as the mixture m, are fed to a second sound classifier to extract embeddings for the source estimates and the mixtureV (2) all . These embeddings condition a second source separation subnetwork TDCN++ (2) which produces the final source estimatesŝ (2) .…”

“…A fundamental problem in machine hearing is deconstructing an acoustic mixture into its constituent sounds. This has been done quite successfully for certain classes of sounds, such as separating speech from nonspeech interference [1,2] or speech from speech [3,4,5,6,7]. However, the more general problem of separating arbitrary classes of sound, known as "universal sound separation", has only recently been addressed [8].…”

Consistency-Aware Multi-Channel Speech Enhancement Using Deep Neural Networks

“…However, its result is often inconsistent [15][16][17], and thus the estimated amplitude and phase may change by applying the inverse STFT (iSTFT) and STFT. Although several DNN-based monaural speech enhancement and separation methods improve the performance by considering consistency [18,19], the consistency was not taken into account in DNN-based multi-channel speech enhancement.…”