Emotion-Based End-to-End Matching Between Image and Music in Valence-Arousal Space

Abstract

Both images and music can convey rich semantics and are widely used to induce specific emotions. Matching images and music with similar emotions might help to make emotion perceptions more vivid and stronger. Existing emotion-based image and music matching methods either employ limited categorical emotion states which cannot well reflect the complexity and subtlety of emotions, or train the matching model using an impractical multi-stage pipeline.In this paper, we study end-to-end matching between image and music based on emotions in the continuous valence-arousal (VA) space. First, we construct a large-scale dataset, termed Image-Music-Emotion-Matching-Net (IMEMNet), with over 140K image-music pairs. Second, we propose cross-modal deep continuous metric learning (CDCML) to learn a shared latent embedding space which preserves the cross-modal similarity relationship in the continuous matching space. Finally, we refine the embedding space by further preserving the single-modal emotion relationship in the VA spaces of both images and music. The metric learning in the embedding space and task regression in the label space are jointly optimized for both cross-modal matching and single-modal VA prediction. The extensive experiments conducted on IMEMNet demonstrate the superiority of CDCML for emotion-based image and music matching as compared to the state-of-the-art approaches. CCS CONCEPTS• Information systems → Sentiment analysis; Multimedia information systems.

Visual Signatures for Music Mood and Timbre

“…Similarly, in [4] the music mood is denoted as the Valence-Arousal value as 2D vectors, which is then mapped to a specific RGB value on a color wheel. In [5], [6], and [7], the moods of both music and images are extracted and compared. Therefore, they allow for the selection of the most relavant music-image pairs within a finite library of music and images.…”

“…Inspired by this encoder-decoder-based structure, we also provided a similar solution in [9]. We re-implemented the image-music mapping model proposed in [6] to construct a dataset with corresponding music and landscape pairs. Following that, we implemented a similar encoder-decoder structure with ResNet50 [13] and StyleGAN3 [14] pre-trained on a landscape dataset [15].…”

Affective Image Content Analysis: Two Decades Review and New Perspectives

“…The system can achieve the retrieval between Chinese folk music and Chinese folk image based on their involved emotions. Chen et al [172] and Zhao et al [173] designed a system that computes the emotional similarity between music and images. With this system, users can generate the mood-aware music slide shows from their personal album photos.…”

ImageFlowing-Enhance Emotional Expression by Reproducing the Vital Signs of the Photographer

“…Matching images and music with similar emotions might help to make emotion perceptions more vivid and stronger. [28] There is a team which propose to musicalize images based on their emotions. The extract visual features inspired by the concept of principles-of-art can recognize image emotions.…”