Audio-Visual Embedding for Cross-Modal Music Video Retrieval through Supervised Deep CCA

Abstract

Deep learning has successfully shown excellent performance in learning joint representations between different data modalities. Unfortunately, little research focuses on cross-modal correlation learning where temporal structures of different data modalities, such as audio and video, should be taken into account. Music video retrieval by a given musical audio is a natural way to search and interact with music contents. In this work, we study cross-modal music video retrieval in terms of emotion similarity. Particularly, an audio of an arbitrary length is used to retrieve a longer or full-length music video. To this end, we propose a novel audio-visual embedding algorithm by Supervised Deep Canonical Correlation Analysis (S-DCCA) that projects audio and video into a shared space to bridge the semantic gap between audio and video. This also preserves the similarity among audio and visual contents from different videos with the same class label and the temporal structure. The contribution of our approach is mainly manifested in the two aspects: i) We propose to select top k audio chunks by attention-based Long Short-Term Memory (LSTM) model, which can represent good audio summarization with local properties. ii) We propose an end-to-end deep model for crossmodal audio-visual learning where S-DCCA is trained to learn the semantic correlation between audio and visual modalities. Due to the lack of music video dataset, we construct 10K music video dataset from YouTube 8M dataset. Some promising results such as MAP and precision-recall show that our proposed model can be applied to music video retrieval.

Embedding-based Music Emotion Recognition Using Composite Loss

“…Cross-modal recognition: Cross-modal recognition approaches using embedding have attracted much attention as a technique that can perform effective bidirectional recognition between different modalities (e.g., image, text and audio). Related to audio processing, some researchers explored cross-modal recognition between audio and image [42], [43] and the one between audio and text (lyrics) [44]. But, to the best of our knowledge, no existing work addresses crossmodal recognition between audio and emotion except our previous study [45], where MultiLayer Perceptrons (MLPs) based on CCA loss are used to compute music and emotion embeddings.…”

Cross-Modal Music-Video Recommendation: A Study of Design Choices

“…Supervised approaches: In the case of supervised learning, the matching criterion that associates the audio and video modalities is deduced from additional sources of information. Typically, mood tags [26] [33] or projections into the valence-arousal plane [23] can be used to recommend musics and videos that have a similar emotional content. The use of mood information accelerates the training, and allows the systems to reach promising retrieval performances.…”

“…Examples of systems for music recommendation given video as input are [11], [12], [23], [26], [27]. In a symmetrical way, examples of systems for video recommendation given audio as input are [13], [33].…”

“…Music recommendation for video has applications in automatic video editing [26], as well as professional music supervision [10]. Video recommendation for music has applications in automatic MTV generation [13] and music recommendation [33].…”

Automatic Assessment of Dysarthric Severity Level Using Audio-Video Cross-Modal Approach in Deep Learning

“…Zheng et al implemented a cross-modal of an audio-video embedding algorithm through Supervised Deep Canonical Correlation Analysis (S-DCCA)[101]. In this model, audio and video are projected into a shared area to address the semantic distance between audio and video.…”