Multimedia content analysis for emotional characterization of music video clips

Abstract

Nowadays, tags play an important role in the search and retrieval process in multimedia content sharing social networks. As the amount of multimedia contents explosively increases, it is a challenging problem to find a content that will be appealing to the users. Furthermore, the retrieval of multimedia contents, which can match users' current mood or affective state, can be of great interest. One approach to indexing multimedia contents is to determine the potential affective state, which they can induce in users. In this paper, multimedia content analysis is performed to extract affective audio and visual cues from different music video clips. Furthermore, several fusion techniques are used to combine the information extracted from the audio and video contents of music video clips. We show that using the proposed methodology, a relatively high performance (up to 90%) of affect recognition is obtained.

Video Affective Content Analysis: A Survey of State-of-the-Art Methods

“…Shot is an important film element that can control a video's tempo. Shot-related features [14], [11], [22], [48], [21], [49], [50] include shot duration, shot transition, and shot type transition. Shot duration represents the length of a shot.…”

“…Typical color features include color saturation, dominant color, color layout, color energy, color heat, color activity, and color weight [18], [19], [27], [43], [24], [14], [8], [9], [8], [10], [11], [21], [49], [50]. Color saturation refers to the intensity of color.…”

“…Widely used music features include dynamics, timbre, harmony, and rhythm. In addition, many audio features are shared among different audio types including energy related features [19], [27], [38], [44], [24], [46], [2], [47], [10], [11], [15], [22], [23], [50], [31], ZCR [19], [27], [38], [11], [15], [49], [50], [31], MFCC [19], [27], [24], [11], [15], [23], [49], [50], spectral centroid [27], [24], [50], spectral rolloff [19], [27], [30], [11], spectral flux [19], [27], [30], [15], etc. These features are used for both emotional dimension prediction and emotional category classification.…”

“…The lighting keys [27], [38], [24], [11], [15], [48], [49], [31] are related to two major aesthetic lighting techniques, i.e., chiaroscuro and flat lighting. The former is characterized by strong contrast between light and shadowed areas, and the later reduces the emphasis of the contrast between light and dark.…”

“…Many machine learning methods have been investigated to model the mapping between video features and discrete emotional descriptors, including support vector machines (SVMs) [80], multi-layer feed-forward neural networks (NNs) [22], Adaboost [8], Gaussian Mixture Models (GMMs) [49], K-Nearest Neighbor (KNN) [50], Hidden Markov Models (HMMs) [10], [61], Dynamic Bayesian Networks (DBNs) [81], and Conditional Random Fields (CRFs) [48]. A classifier is divided into static or dynamic based on temporal information.…”