Advancing naturalistic affective science with deep learning

Abstract

People express emotions via a variety of behaviors, including facial muscle movements, body poses and gestures, vocal prosody, and speech. To understand how people experience and perceive emotion, it is crucial to quantify and model these behaviors. However, existing methods are insufficient to address this need. Manually annotating behavior is very time-consuming, making it infeasible to do at scale. Moreover, common linear models cannot fully capture the complex, nonlinear, and interactive affective processes embodied by these behaviors. In this methodology review, we describe how deep learning addresses these challenges and thereby promises to advance naturalistic affective science. First, deep learning provides accessible and efficient tools to annotate dynamic, complex, multi-modal behaviors. These automated annotation tools can scale up behavioral quantification to a degree impossible with human coders, enabling many new, more naturalistic approaches to affective science. Second, deep learning offers innovative paradigms for optimizing and manipulating naturalistic stimuli. This application makes it possible to generate experiment designs with greater generalizability, statistical power, and external validity. Third, deep learning can support flexible, powerful cognitive models of naturalistic affective processing. These novel cognitive models make it possible to explain how the mind and brain engage in the emotional world in ways that are both broader and more precise. However, deep learning is not without its limitations, so we also explore important failure cases, practical issues, and ethical concerns. By detailing the promise and the peril of deep learning, this review paves the way for a more naturalistic and generalizable affective science.

Multimodal investigations of emotional face processing and social trait judgment of faces

“…In a similar vein, a face is perceived to belong to someone with high competence if the face is accompanied by richer than poorer clothes 168 . To address this limitation, recent research has incorporated dynamic facial movement and naturalistic face images as stimuli to study the perception of affective states 163,169 and social traits 114,118,170 …”

“…More naturalistic stimuli, such as photographs of individuals from diverse races with varied facial expressions and in complex contexts, should be used. Notably, deep neural networks can effectively analyze such naturalistic faces 170 . Specifically for studying social trait judgments, the third recommendation is to sample comprehensive sets of social traits and faces for collecting human judgments 118 .…”

“…168 To address this limitation, recent research has incorporated dynamic facial movement and naturalistic face images as stimuli to study the perception of affective states 163,169 and social traits. 114,118,170 Combining multimodal neuroscience approaches with these new ways of probing the naturalistic affective and social judgment processes will shed new lights on the understanding of the neurobiological bases of emotional face processing and social trait judgment of faces.…”

“…Notably, deep neural networks can effectively analyze such naturalistic faces. 170 Specifically for studying social trait judgments, the third rec-ommendation is to sample comprehensive sets of social traits and faces for collecting human judgments. 118 Prior research has examined a limited number of social traits, which may not be representative of the full range of trait judgments that people make from faces in a wide range of contexts.…”

Emotion prediction as computation over a generative theory of mind

“…Other computer science work favours training less-structured models. This approach forgoes extensive hand-coding, opting to learn latent representations from statistical regularities in large datasets [78][79][80], see [81][82][83] for reviews. These learned latent representations can encode patterns mapping between emotion labels and expressions, scenes, objects, actions and social interactions [84][85][86][87][88].…”