This Music Reminds Me of a Movie, or Is It an Old Song? An Interactive Audiovisual Journey to Find out, Explore and Play

Towards Multimodal Search and Visualization of Movies Based on Emotions

“…stories), and by the user emotional expressions detected with a camera (described above), one can search by a music being played (Figure 7). This extends the features in As Music Goes By [20,21,33], allowing to search by selecting a mic icon, getting the music detected, searching for music with similar emotional impact; and from the movies tab in this music page (Figure 7b), accessing movies where that music is featured, where this music can be highlighted in the corresponding soundtrack in synchrony with the movie. In short, this music unexpectedly being played in the current environment could take the users by serendipitous browsing into a movie scene where this music is also played, and that could end up being quite valuable to them.…”

“…The users' emotional impact is being assessed while users watch the movies, to provide user feedback and to catalog the movies: based on biosensors like EEG, EDA, and a webcam for facial expressions; and by having users engaging in self-assessment and annotation of the movies, using different models or interfaces like categorical emotions, maniken and emotional wheel (both based on the VA dimensions) [23]; and articulating with other project tasks where video content-based features are extracted mostly in audio, subtitles and image. This application is also integrating our previous As Music Goes By [20,21,33], allowing users to search, visualize and explore music and movies from complementary perspectives of music versions, artists, quotes and movie soundtracks. Next, we present and discuss our emotional model approach, that we want to keep rich and expressive, though effective, flexible and easy to understand; and the movie visualization and search features based on their emotional impact, as a whole and along time, with multimodal interfaces for different contexts of use.…”

“…In addition to highlighting the emotion associated with that song (one per song, as registered in Spotify), a button is displayed beneath to search for music with similar emotional impact, as exemplified in Figure 7 c). , From the movies tab in this music page (b), users can also access movies where that music is featured [20,21], giving access to the movie page, where this music can be highlighted in the corresponding soundtrack that could be accessed in synchrony with the movie; c) songs with similar emotions: the emotion wheels highlight the associated emotions. stories), and by the user emotional expressions detected with a camera (described above), one can search by a music being played (Figure 7).…”

“…Such situations may be when we want our current emotion to be taken into account, or want to draw an emotional path to search for, or when we are listening to a music that moves us and we do not identify it, nor the movie it reminds us of. Music has always played a significant role in movies, even when they were silent, with significant emotional impact [20,21], and it can also be a trigger to get us into one of them in such situations, in serendipitous moments.…”