A Multimodal Database for Affect Recognition and Implicit Tagging

Abstract

Abstract-MAHNOB-HCI is a multimodal database recorded in response to affective stimuli with the goal of emotion recognition and implicit tagging research. A multimodal setup was arranged for synchronized recording of face videos, audio signals, eye gaze data, and peripheral/central nervous system physiological signals. Twenty-seven participants from both genders and different cultural backgrounds participated in two experiments. In the first experiment, they watched 20 emotional videos and self-reported their felt emotions using arousal, valence, dominance, and predictability as well as emotional keywords. In the second experiment, short videos and images were shown once without any tag and then with correct or incorrect tags. Agreement or disagreement with the displayed tags was assessed by the participants. The recorded videos and bodily responses were segmented and stored in a database. The database is made available to the academic community via a web-based system. The collected data were analyzed and single modality and modality fusion results for both emotion recognition and implicit tagging experiments are reported. These results show the potential uses of the recorded modalities and the significance of the emotion elicitation protocol.

A survey of affective brain computer interfaces: principles, state-of-the-art, and challenges

“…Research toward such neurophysiologybased implicit tagging approaches of multimedia content has suggested its feasibility. [21,22] Liberati et al [23] showed that aBCI can be used for the communication of emotional responses toward a certain object. They 27] However, such stimulus-independent passive approaches (see below) might turn into active approaches, for example when players realize that their affective state has an influence on game parameters, and therefore begin to self-induce states to manipulate the gaming environment according to their preferences.…”

Emotion Understanding Using Multimodal Information Based on Autobiographical Memories for Alzheimer’s Patients

“…Therefore, the results are not reliable on naturalistic conditions regarding illumination, headpose variations and nature of expressions. Nevertheless, there are non-posed datasets to test naturalistic expressions such as SEMAINE [23], MAHNOB-HCI [21] or DECAF [52]. In these cases the illumination and head-pose variation problems are taken into account depending on the aim of the study.…”

“…The work in [20] considers both types of data using the MAHNOB-HCI database [21]. The EEG signal was captured using 32 sensors and the power spectral density was extracted from overlapping one second windows.…”

“…Several datasets, focusing on different applications, are available for emotion recognition. For example, DEAP dataset provides EEG and face recordings of participants while they watch musical videos just for the analysis of human affective states [9]; SEMAINE database aims to provide voice and facial information to study the behaviour of subjects interacting with virtual avatars [23]; MAHNOB-HCI database was created for the study of emotions while humans are watching multimedia, supplying several data such as audio, an RGB video and five monochrome videos of the face, EEG, ECG, respiration amplitude, skin temperature and eye-gaze data [21]; or CASMEII dataset which studies facial micro-expressions for security and medical applications, requiring cameras of higher frame rate and spatial resolution [24].…”

Human-centered implicit tagging: Overview and perspectives

“…The users' behavior and spontaneous reactions to multimedia data can provide useful information for multimedia indexing with the following scenarios: (i) direct assessment of tags: users spontaneous reactions will be translated into emotional keywords, e.g., funny, disgusting, scary [3], [4], [5], [6]; (ii) assessing the correctness of explicit tags or topic relevance, e.g., agreement or disagreement over a displayed tag or the relevance of the retrieved result [7], [8], [9], [10]; (iii) user profiling: a user's personal preferences can be detected based on her reactions to retrieved data and be used for re-ranking the results; (iv) content summarization: highlight detection is also possible using implicit feedbacks from the users [11], [12].…”