The role of temporal inversion in the perception of realistic and morphed dynamic transitions between facial expressions

Abstract

Recent studies suggest that video recordings of human facial expressions are perceived differently than linear morphing between the first and last frames of these records. Also, observers can differentiate dynamic expressions presented in normal versus time-reversed frame orders. To date, the simultaneous influence of dynamics (natural or linear) and timeline (normal or reversed) has not yet been tested on a wide range of dynamic emotional expressions and the transitions between them. We compared the perception of dynamic transitions between basic emotions in realistic (human-posed) and artificial (linearly morphed) stimuli which were presented in reversed or non-reversed order. The nonlinearity of realistic stimuli was demonstrated by automated facial structure analysis. The results of the behavioral study revealed that the recognition of emotions in time-reversed stimuli significantly differed from recognition of the normally presented ones, and this difference was substantially higher for videos of a dynamic human face than for linear morphs. Emotions displayed at the end of the transitions were recognized better than the first-frame emotions in all types of stimuli except in the time-reversed videos, which showed a similar recognition rate for both the starting and ending emotions. Our findings suggest that nonlinearity, which is present in a realistic facial display but absent in linear morphing, is an important cue for emotion perception, and that unnatural perceptual conditions (inversion in time) make the recognition of emotions more difficult. These results confirm the ability of the human visual system to use subtle dynamic cues on an interlocutor's face, and reveal its sensitivity to the timeline organization of the displayed emotions.

Use and Usefulness of Dynamic Face Stimuli for Face Perception Studies—a Review of Behavioral Findings and Methodology

“…Subjects consistently selected the animations closer to natural motion, demonstrating high sensitivity to deviations from natural motion. In line with these results, face stimuli based on motion created by linear morphing techniques (e.g., linear morphing between two frames) can lead to less accurate emotion recognition (Wallraven et al, 2008 ; Cosker et al, 2010 ; Korolkova, 2018 ) and are often perceived as less natural (Cosker et al, 2010 ) than natural motion. Moreover, humans are sensitive to specific properties of natural motion (e.g., velocity; Pollick et al, 2003 ; Hill et al, 2005 ; Bould et al, 2008 ), to temporal sequencing (e.g., temporal asymmetries in the unfolding of facial expressions; Cunningham and Wallraven, 2009 ; Reinl and Bartels, 2015 ; Delis et al, 2016 ; Korolkova, 2018 ) and even to perceptual interactions between dynamic facial features (e.g., eye and mouth moving together during yawning; Cook et al, 2015 ).…”

“…Videos have been used to investigate neural representations of emotional valence that generalize across different types of stimuli (Skerry and Saxe, 2014 ; Kliemann et al, 2018 ). Other studies have manipulated the order of video frames to investigate the importance of the temporal unfolding of facial expressions (Cunningham and Wallraven, 2009 ; Reinl and Bartels, 2015 ; Korolkova, 2018 ), or the neural sensitivity to natural facial motion dynamics (Schultz and Pilz, 2009 ; Schultz et al, 2013 ). While for these research questions, videos of faces achieved a good balance between ecological validity and experimental control, the content of information in such videos is technically challenging to assess (compare “photo-realistic face rendering” below), let alone to parametrically control it.…”

“…Such stimuli have been used to compare the recognition thresholds for static and dynamic faces (Calvo et al, 2016 ) or the perception of the intensity of facial expressions (Recio et al, 2014 ). Despite these useful findings, such stimuli represent solely a coarse linear approximation of natural face motion, which might lead to less accurate emotion recognition than their natural counterparts (Wallraven et al, 2008 ; Cosker et al, 2010 ; Korolkova, 2018 ). Moreover, these stimuli do not allow separating form and motion information, which is necessary to investigate identity-from-motion for example.…”

“…These recent developments bear potential for face perception research. First, new developments reduce the technical demands of recording facial movements allowing markerless tracking by using for example depth sensors (e.g., Walder et al, 2009 ; Girges et al, 2015 ), automated landmark detection (Korolkova, 2018 ), or simply RGB channels in videos (Thies et al, 2016 ). Second, recent facial animation and machine learning advancements (e.g., deep learning) now allow creating naturalistic dynamic face stimuli indistinguishable from real videos (e.g., Thies et al, 2016 ; Suwajanakorn et al, 2017 ).…”

Perception of “Live” Facial Expressions

“…Например, компьютерная анимация, при которой изменения лица происходят по кратчайшей прямой, приводит к замедленному и менее точному распознаванию эмоций, а также к более низким оценкам интенсивности, искренности и типичности по сравнению с действительным развертыванием экспрессии [38; 130]. В силу нелинейности и асинхронии активности мимических мышц при переживании эмоции [88], естественные динамические экспрессии и переходы между ними воспринимаются иначе, чем искусственные динамические последовательности, созданные путем линейного морфинга [81], и приводят к различиям в волновой активности головного мозга [103]. Точность категоризации снижается при инверсии естественных видеофрагментов во времени [81; 107].…”