Context in Human Emotion Perception for Automatic Affect Detection: A Survey of Audiovisual Databases

Abstract

An important aspect of human emotion perception is the use of contextual information to understand others' feelings even in situations where their behavior is not very expressive or has an emotionally ambiguous meaning. For technology to successfully detect affect, it must mimic this human ability when analyzing audiovisual input. Databases upon which machine learning algorithms are trained should capture the context of social interactions as well as the behavior expressed in them. However, there is a lack of consensus about what constitutes relevant context in such databases. In this article, we make two contributions towards overcoming this challenge: (a) we identify two principal sources of context for emotion perceptions based on psychological theory, and (b) we provide an overview of how each of these has been considered in published databases covering social interactions. Our results show that a similar set of contextual features are present across the reviewed databases. Between all the different databases researchers seem to have taken into account a set of contextual features reflecting the sources of context seen in psychological theory. However, within individual databases, these features are not yet systematically varied. This is problematic because it prevents them from being used directly as resources for the modeling of context-sensitive affect detection. Based on our findings, we suggest improvements for the future development of affective databases.

Automatic Emotion Recognition for Groups: A Review

“…Ideally, data should be annotated without the annotation process influencing the (labelled) data, and collected without the collecting itself influencing the data. The former issue was raised in a recent review by Dudzik et al [87], describing how an annotator (perceiver of the emotion data) can be biased in his or her interpretation of other people's emotions. Regarding the latter issue, an example of the ideal situation could be to search for social media messages related to an event, where the collection itself will not influence the content of these messages.…”

Exploring Personal Memories and Video Content as Context for Facial Behavior in Predictions of Video-Induced Emotions

“…Such information about triggering events has a strong role in interpreting facial behavior [42]. Affective detection work has only tentatively explored this aspect because it is conceptually challenging to translate into automatic systems and generally lacks available corpora for modeling [23].…”

“…The insights gained by this act of emotional perspective-taking can complement any information offered by behavior in isolation, thereby enabling an observer to make accurate inferences even for ambiguous cases (e.g., [41]). However, context-sensitive approaches remain under-explored in automatic affect detection [23], despite researchers generally acknowledging their potential [60,66,68]. Likely causes for this neglect are the substantial challenges involved in (1) identifying relevant contextual influences for emotional responses in an application setting, as well as (2) developing technical solutions that provide automatic systems with an awareness of them [29].…”

“…Likely causes for this neglect are the substantial challenges involved in (1) identifying relevant contextual influences for emotional responses in an application setting, as well as (2) developing technical solutions that provide automatic systems with an awareness of them [29]. Overcoming these challenges requires systematic exploration of person-and situation-specific influences in computational modeling activities [23] informed by findings from the social sciences [3]. Compared to emotional responses in general, situations in which video stimuli are consumed by an individual provide a more constrained scenario for the exploration of relevant contextual influences.…”

Personal data protection and academia: GDPR issues and multi-modal data-collections "in the wild"

“…Simply speaking, academia is aiming for "in the wild" data collections, meaning, to process information of people even when they are not aware of it. This entails the use of data enriched with additional metadata such as age, sex, profession, socio-demographic information (Dudzik et al, 2019), or specific personality traits (similar to Big Data studies that use freely available data found on the Internet). We consider this type of data (e.g., sex, age, language, proficiency, personality, etc.)…”