Dilated U-net based approach for multichannel speech enhancement from First-Order Ambisonics recordings

Abstract

We present a CNN architecture for speech enhancement from multichannel first-order Ambisonics mixtures. The data-dependent spatial filters, deduced from a mask-based approach, are used to help an automatic speech recognition engine to face adverse conditions of reverberation and competitive speakers. The mask predictions are provided by a neural network, fed with rough estimations of speech and noise amplitude spectra, under the assumption of known directions of arrival. This study evaluates the replacing of the recurrent LSTM network previously investigated by a convolutive U-net under more stressing conditions with an additional second competitive speaker. We show that, due to more accurate short-term masks prediction, the U-net architecture brings some improvements in terms of word error rate. Moreover, results indicate that the use of dilated convolutive layers is beneficial in difficult situations with two interfering speakers, and/or where the target and interferences are close to each other in terms of the angular distance. Moreover, these results come with a two-fold reduction in the number of parameters.

L3DAS21 Challenge: Machine Learning for 3D Audio Signal Processing

“…For Task 1 (SE), we use a Filter and Sum Network architecture (FaSNet) [5], adapted from this public PyTorch implementation 8 . This network is a state-of-the-art neural beamformer that operates in the time domain and, therefore, work on both the magnitude and the phase information of the signal.…”

“…Neural beamforming techniques as Filter and Sum Networks (FaS-Net) [5] provide state-of-the art results for Ambisonics-based SE and are usually suitable for low-latency scenarios. Also U-Net-based approaches provide competitive results in this context, both for monaural [6,7] and multichannel SE tasks [8], at the expense of higher computational power demand. Other techniques to perform SE include recurrent neural networks (RNNs) [9], graph-based spectral subtraction [10], discriminative learning [11], dilated convolutions [12,13].…”

Towards speech enhancement using a variational U-Net architecture

“…The U-Net architecture [16], originally proposed for biomedical signal processing, has recently been used for audio source separation tasks [14,17,18]. More importantly, it has also been adapted for speech enhancement [11] with the addition of dilated convolutions. It has been shown that the generation of filter masks with a dilated U-Net performed better than without dilation, indicating the usefulness of dilation in arXiv:2012.03594v1 [eess.AS] 7 Dec 2020 such a network.…”

“…Deep networks predominantly use higher-dimension log-power spectra with a comparably long temporal context in an attempt to learn features best representing clean speech [9,10]. Recent works focus on using DNNs to generate ideal filter masks based on these self-learned speech representations [11,12], where each mask corresponds to one target speaker. Psycho-acoustic modeling is combined with a speech quality optimization target in [13].…”

L3DAS21 Challenge: Machine Learning for 3D Audio Signal Processing

“…Neural beamforming techniques as Filter and Sum Networks (FaS-Net) [5] provide state-of-the art results for Ambisonics-based SE and are usually suitable for low-latency scenarios. Also U-Net-based approaches provide competitive results in this context, both for monaural [6,7] and multichannel SE tasks [8], at the expense of higher computational power demand. Other techniques to perform SE include recurrent neural networks (RNNs) [9], graph-based spectral subtraction [10], discriminative learning [11], dilated convolutions [12].…”