Evaluating Audiovisual Source Separation in the Context of Video Conferencing

Abstract

Source separation involving mono-channel audio is a challenging problem, in particular for speech separation where source contributions overlap both in time and frequency. This task is of high interest for applications such as video conferencing. Recent progress in machine learning has shown that the combination of visual cues, coming from the video, can increase the source separation performance. Starting from a recently designed deep neural network, we assess its ability and robustness to separate the visible speakers' speech from other interfering speeches or signals. We test it for different configuration of video recordings where the speaker's face may not be fully visible. We also asses the performance of the network with respect to different sets of visual features from the speakers' faces.

Deep-learning-based audio-visual speech enhancement in presence of Lombard effect

“…Estimators of speech SII [110] 1997 Used for additive stationary noise or [108] intelligibility bandwidth reduction CSII [130] 2004 Extension of SII for broadband peak- [108] clipping and center-clipping distortion ESII [210] 2005 Extension of SII for fluctuating noise [108] STOI [241] 2011 Able to predict quite accurately speech [7], [37], [55], [77], [85], [108], [109] intelligibility in several situations [99], [122], [128], [136] HASPI [132] 2014 Specifically designed for hearing- [99], [100] impaired listeners ESTOI [124] 2016 Extension of STOI for highly [107], [108], [176], [178], [179], [244] optimally performed, because floor or ceiling effects might occur if the listeners' task is too hard or too easy. This issue can be mitigated by testing the system at several SNR within a pre-determined range, at the expense of the time needed to conduct the listening experiments.…”

“…This means that in order to have good performance in a wide variety of settings, very large AV datasets for training and testing need to be collected. In practice, the systems are trained using a large number of complex acoustic [66], [76], [77], [85], [99], [122], [128], [164], [165], [176], [178], [179], [220]- [222], [244], [263], [274], [ [17], [65], [154], [164], [165] Landmark-based features [100], [154], [183], [203] Multisensory features [195] Face recognition embedding [55], [109], [169], [192], [239] VSR embedding [7], [10], [107]- [109], [153], [222], [273] Facial appearance embedding [42], [208] Compressed mouth frames [37] Speaker direction [85], [244], [279] Acoustic Features…”

“…Magnitude spectrogram [3]- [7], [10], [12], [17], [37], [42], [65], [66], [76], [77], [85], [99], [100], [107], [122], [128], [136], [153], [154], [164] [165], [176], [178], [179], [183], [192], [195], [203], [208], [220]- [222], [244], [263], [274], [279] Phase a [7], [10], [153] Complex spectrogram [55], [107], [109], [169], [239] Raw waveform [108], [273] Speaker embeddings [10], [85], [169], [192], [ [6],…”

“…The features extracted from such a network provided an AV representation that allows to achieve superior performance compared to an AO-SE approach. Besides multisensory features, embeddings extracted with face recognition [55] or VSR [7] models have been shown to be effective.İnan et al [109] performed a study to evaluate the differences between these two kinds of embeddings. Their results showed that VSR embeddings were able to separate voice activity and silence regions better than face recognition embeddings, which could provide a better distinction between speakers instead.…”

“…Recent works have used as acoustic input to the AV system either the magnitude spectrogram and the respective phase [7], [10], [153], the real and the imaginary parts of the complex spectrogram [55], [107], [109], [169], [239], or directly the raw waveform [108], [273]. Although these approaches allow to incorporate and process the full information of an acoustic signal, research in this area is still active and suggests that there is still room for improvement by exploiting the full information of the noisy speech signal [168], [281].…”