Deep Latent Space Learning for Cross-Modal Mapping of Audio and Visual Signals

Abstract

We propose a novel deep training algorithm for joint representation of audio and visual information which consists of a single stream network (SSNet) coupled with a novel loss function to learn a shared deep latent space representation of multimodal information. The proposed framework characterizes the shared latent space by leveraging the class centers which helps to eliminate the need of pairwise or triplet supervision. We quantitatively and qualitatively evaluate the proposed approach on VoxCeleb, a benchmarks audio-visual dataset on multitude of tasks including cross-modal verification, cross-modal matching and cross-modal retrieval. State-of-the-art performance is achieved on cross-modal verification and matching while comparable results are observed on the remaining applications. Our experiments demonstrate the effectiveness of the technique for cross-modal biometric applications.

A Multi-View Approach to Audio-Visual Speaker Verification

“…Cross-modal processing has been recently used in different combinations such as audio-video [15,14,16,17] and speech-text [18]. The common approach in these studies is to map inputs from different modalities into a shared space to achieve cross-modal retrieval.…”

“…In [16], same-different classification is performed on the cosine scores between face and voice embeddings to train the system. In [17], a novel loss function is proposed to learn the embeddings in a shared space. Their loss function tries preserving neighborhood constraints within and across modalities.…”

“…As described in Section 4, the main goal of the multi-view model is to do cross-modal testing. We simulate this A vs. Test pairs VC2 EER VC1 EER A vs V of [15] NA 29.5 A vs V of [17] NA 29.6 A vs V of [16] 22.5 NA A vs V (our) 29.5 28.0 [25]). Since A vs. V cross-modal verification setting is the most difficult situation, it has higher EER as compared to Table 2 but it is still better than the 50% chance level.…”

A Multi-View Approach To Audio-Visual Speaker Verification

“…Cross-modal processing has been recently used in different combinations such as audio-video [15,14,16,17] and speech-text [18]. The common approach in these studies is to map inputs from different modalities into a shared space to achieve cross-modal retrieval.…”

“…In [16], same-different classification is performed on the cosine scores between face and voice embeddings to train the system. In [17], a novel loss function is proposed to learn the embeddings in a shared space. Their loss function tries preserving neighborhood constraints within and across modalities.…”