A Multi-View Approach to Audio-Visual Speaker Verification

Abstract

Although speaker verification has conventionally been an audio-only task, some practical applications provide both audio and visual streams of input. In these cases, the visual stream provides complementary information and can often be leveraged in conjunction with the acoustics of speech to improve verification performance. In this study, we explore audio-visual approaches to speaker verification, starting with standard fusion techniques to learn joint audio-visual (AV) embeddings, and then propose a novel approach to handle cross-modal verification at test time. Specifically, we investigate unimodal and concatenation based AV fusion and report the lowest AV equal error rate (EER) of 0.7% on the VoxCeleb1 dataset using our best system. As these methods lack the ability to do cross-modal verification, we introduce a multi-view model which uses a shared classifier to map audio and video into the same space. This new approach achieves 28% EER on VoxCeleb1 in the challenging testing condition of cross-modal verification.

Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT

“…As is expected, fine-tuning with a large amount of labeled data improves performance. In audio-visual setting, our best model (0.84%) outperforms [33] (1.8% and 1.4%) with a single model and slightly falls behind its ensembled model (0.7%). Note in contrast to the prior works [18,33,32] which uses the whole face, our model only relies on the lip area of the speaker as visual input and achieves a better trade-off between privacy and performance.…”

“…In audio-visual setting, our best model (0.84%) outperforms [33] (1.8% and 1.4%) with a single model and slightly falls behind its ensembled model (0.7%). Note in contrast to the prior works [18,33,32] which uses the whole face, our model only relies on the lip area of the speaker as visual input and achieves a better trade-off between privacy and performance. In addition, we acknowledge the gap between our best model and the current SOTA on VC1 ( [25]: 0.38%).…”

“…We should note that unlike most of the prior work on learning audio-visual speaker embeddings [32,33], we use lip video (see figure 2) instead of whole-face video or image as input. This can significantly improve the robustness in noisy environments compared to audio-based systems while reducing the amount of biometric information required compared to facebased systems.…”

“…We report SC accuracy (SC-ACC) and SV-EER on VC1 using the official test set and test trials. As for VC2-EER, since there does not exist official test trials, we follow [33] to create one by sampling one positive trial and one negative trial for each test set utterance. To probe noise robustness, we follow [16] to create 20 noisy test sets for VC1 and VC2, where each clean set is mixed with {Babble, Speech, Music, Other} noise at a SNR in {-10, -5, 0, 5, 10} dB.…”

Cross-Modal Perceptionist: Can Face Geometry be Gleaned from Voices?

“…These tasks are inherently selection problems in which the best fit of a voice-face pair from the dataset is desired. Another similar task is cross-modal verification [32,48,52] that tells whether input faces and voices belong to the same person, which is a simply classification problem for paired inputs. Our work solves its root question and explains the success in voice-face matching or verification by verifying correlations between voices and face geometry.…”