Using automated computer vision and machine learning to code facial expressions of affect and arousal: Implications for emotion dysregulation research

Abstract

As early as infancy, caregivers’ facial expressions shape children's behaviors, help them regulate their emotions, and encourage or dissuade their interpersonal agency. In childhood and adolescence, proficiencies in producing and decoding facial expressions promote social competence, whereas deficiencies characterize several forms of psychopathology. To date, however, studying facial expressions has been hampered by the labor-intensive, time-consuming nature of human coding. We describe a partial solution: automated facial expression coding (AFEC), which combines computer vision and machine learning to code facial expressions in real time. Although AFEC cannot capture the full complexity of human emotion, it codes positive affect, negative affect, and arousal—core Research Domain Criteria constructs—as accurately as humans, and it characterizes emotion dysregulation with greater specificity than other objective measures such as autonomic responding. We provide an example in which we use AFEC to evaluate emotion dynamics in mother–daughter dyads engaged in conflict. Among other findings, AFEC (a) shows convergent validity with a validated human coding scheme, (b) distinguishes among risk groups, and (c) detects developmental increases in positive dyadic affect correspondence as teen daughters age. Although more research is needed to realize the full potential of AFEC, findings demonstrate its current utility in research on emotion dysregulation.

Digital Behavioral Phenotyping Detects Atypical Pattern of Facial Expression in Toddlers with Autism

“…In recent years, there has been progress on automatic facial expression analysis of both children and toddlers [Dys & Malti, 2016; Gadea, Aliño, Espert, & Salvador, 2015; Haines et al, 2019; LoBue & Thrasher, 2014; Messinger, Mahoor, Chow, & Cohn, 2009]. In addition to this, we have previously validated our CVA algorithm against expert human rater coding of facial affect in a subsample of 99 video recordings across 33 participants (ASD = 15, non‐ASD = 18).…”

Facial Expression Analysis Based on Fusion Multi-Layer Convolutional Layer Feature Neural Network

“…But there are still many pacts need to be improved. (1)In this paper, only the deepest feature vector and the penultimate layer feature vector in the CNN are used for feature fusion. According to the use of the features of the convolutional layer by the full CNN, we can also try to use the previous layer or even the first two layers of the sub-deep feature vectors.…”

“…Facial expression analysis refers to the use of computers to analyze human facial expressions and changes through pattern recognition and machine learning algorithms and to judge human psychology and emotions, thereby achieving intelligent human-computer interaction [1]. Deep convolution neural network has the outstanding characteristics of unsupervised feature learning, which has been proved to have the ability to mine the deep potential distributed expression features of data in the fields of image, speech and text.…”

A Novel Emotion Control System for Embedded Human–Computer Interaction in Green Iot

“…Elhai et al [21] explored the relationship between human emotional state and the use frequency of smartphones, and designed an emotional adjustment strategy by controlling the use time of smartphones. Haines et al [22] combined computer vision and machine learning to encode facial expressions in real time and studied the emotional effects of facial expressions on receptors. Villani et al [23] evaluated and studied using video games as a treatment tool for patients with emotional disorders such as depression, and exerted emotional interventions on patients through different gaming experiences.…”