Decoding affect in videos employing the MEG brain signal

Abstract

Abstract-This paper presents characterization of affect (valence and arousal) using the Magnetoencephalogram (MEG) brain signal. We attempt single-trial classification of movie and music videos with MEG responses extracted from seven participants. The main findings of this study are that: (i) the MEG signal effectively encodes affective viewer responses, (ii) clip arousal is better predicted than valence employing MEG and (iii) prediction performance is better for movie clips as compared to music videos.

A Multi-task Learning Framework for Time-continuous Emotion Estimation from Crowd Annotations

“…In this section, we attempt to predict the gold standard (or groundtruth) dynamic V/A ratings for each clip from audio-visual features using MTL, and show why learning the audio visual feature- (1) Average energy of audio signal [14] and first pitch frequency Formants (4) Formants up to 4400Hz Time frequency (8) mean and std of: MSpectrum flux, Spectral centroid, Delta spectrum magnitude, Band energy ratio [14] Zero crossing rate (1) Average zero crossing rate of audio signal [14] Silence ratio (2) Mean and std of proportion of silence in a time window [5,14] Video features Description Brightness (6) Mean of: Lighting key, shadow proportion, visual details, grayness, median of Lightness for frames, mean of median saturation for frames Color Features (41) Color variance, 20-bin histograms for hue and lightness in HSV space VisualExcitement (1) Features as defined in [29] Motion (1) Mean inter-frame motion [15] emotion relationship simultaneously for the 12 movie scenes is more effective than learning scene-specific models. Fig.…”

“…In a seminal study affective movie study, Gross et al [6] compiled a collection of movie clips to evoke eight emotional states such as anger, disgust, fear and neutral based on emotion ratings compiled for 250 movie clips from 954 subjects. [2,11,24] are three recent works that have attempted affect recognition from physiological responses of a large population of users to music and movie stimuli.…”

The Secret Language of Our Body

“…A maximum 84% recognition accuracy on six basic emotions is achieved in [11] using GSR, heart rate and temperature signals. Abadi et al [1] compiled a 18 subjects' dataset with Magnetoencephalogram (MEG), Electrooculogram (EOG), Electromyogram (EMG) and ECG responses for 40 music and 36 movie clips, and observed that emotion elicitation (measured on the Arousal-Valence dimensions), and affect prediction, was better for movie clips as compared to music video clips.…”

Video Affective Content Analysis: A Survey of State-of-the-Art Methods

“…By comparing the distribution of facial temperatures to the summarized video clip events from a dataset of 10 subjects viewing three film clips, they concluded that different facial regions exhibit different thermal patterns, and that global temperature changes are consistent with stimulus changes. Abadi et al [95] introduced the magnetoencephalogram (MEG), a non-invasive recording of brain activity, to differentiate between low versus high arousal and low versus high valence using naive Bayes classifier. They collected MEG responses from seven participants watching 32 movie clips and 40 music videos.…”