How do people respond to computer-generated versus human faces? A systematic review and meta-analyses

Miller, Elizabeth J.; Foo, Yong Zhi; Mewton, Paige; Dawel, Amy

doi:10.1016/j.chbr.2023.100283

Cited by 11 publications

(9 citation statements)

References 134 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We would expect a similar pattern if our experiments were repeated with different levels of auditory noise. While our study only examined speech perception, a similar approach could be taken to compare real and synthetic faces in other domains, such as emotions and looking behavior (Miller et al, 2023).…”

Section: Discussionmentioning

confidence: 99%

The Effect on Speech-in-Noise Perception of Real Faces and Synthetic Faces Generated with either Deep Neural Networks or the Facial Action Coding System

Yu,

Lado,

Zhang

et al. 2024

Preprint

View full text Add to dashboard Cite

The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers. We sought to determine whether synthetic talking faces generated by an alternative method would provide a greater perceptual benefit. The facial action coding system (FACS) is a comprehensive system for measuring visually discernible facial movements. Because the action units that comprise FACS are linked to specific muscle groups, synthetic talking faces generated by FACS might have greater verisimilitude than DNN synthetic faces which do not reference an explicit model of the facial musculature. We tested the ability of human observers to identity speech-in-noise accompanied by a blank screen; the real face of the talker; and synthetic talking face generated either by DNN or FACS. We replicated previous findings of a large benefit for seeing the face of a real talker for speech-in-noise perception and a smaller benefit for DNN synthetic faces. FACS faces also improved perception, but only to the same degree as DNN faces. Analysis at the phoneme level showed that the performance of DNN and FACS faces was particularly poor for phonemes that involve interactions between the teeth and lips, such as /f/, /v/, and /th/. Inspection of single video frames revealed that the characteristic visual features for these phonemes were weak or absent in synthetic faces. Modeling the realvs.synthetic difference showed that increasing the realism of a few phonemes could substantially increase the overall perceptual benefit of synthetic faces, providing a roadmap for improving communication in this rapidly developing domain.

show abstract

Section: Discussionmentioning

confidence: 99%

The Effect on Speech-in-Noise Perception of Real Faces and Synthetic Faces Generated with either Deep Neural Networks or the Facial Action Coding System

Yu,

Lado,

Zhang

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Representing realistic and, at the same time, dynamic and responsive human avatars in virtual environments remains a demanding task (Di Natale et al., 2023; Hepperle et al., 2022). Virtual faces created with today's advanced digital graphics technologies, although appearing remarkably realistic, are still perceived as not fully human, which is reflected at both the behavioral and neural levels ( uncanny valley effect ; Di Natale et al., 2023; Hepperle et al., 2022; Miller et al., 2023; Schindler et al., 2017; Sollfrank et al., 2021).…”

Section: Discussionmentioning

confidence: 99%

Electrophysiological correlates of face and object perception: A comparative analysis of 2D laboratory and virtual reality conditions

Sagehorn,

Johnsdorf,

Kisker

et al. 2024

Psychophysiology

View full text Add to dashboard Cite

Human face perception is a specialized visual process with inherent social significance. The neural mechanisms reflecting this intricate cognitive process have evolved in spatially complex and emotionally rich environments. Previous research using VR to transfer an established face perception paradigm to realistic conditions has shown that the functional properties of face‐sensitive neural correlates typically observed in the laboratory are attenuated outside the original modality. The present study builds on these results by comparing the perception of persons and objects under conventional laboratory (PC) and realistic conditions in VR. Adhering to established paradigms, the PC‐ and VR modalities both featured images of persons and cars alongside standard control images. To investigate the individual stages of realistic face processing, response times, the typical face‐sensitive N170 component, and relevant subsequent components (L1, L2; pre‐, post‐response) were analyzed within and between modalities. The between‐modality comparison of response times and component latencies revealed generally faster processing under realistic conditions. However, the obtained N170 latency and amplitude differences showed reduced discriminative capacity under realistic conditions during this early stage. These findings suggest that the effects commonly observed in the lab are specific to monitor‐based presentations. Analyses of later and response‐locked components showed specific neural mechanisms for identification and evaluation are employed when perceiving the stimuli under realistic conditions, reflected in discernible amplitude differences in response to faces and objects beyond the basic perceptual features. Conversely, the results do not provide evidence for comparable stimulus‐specific perceptual processing pathways when viewing pictures of the stimuli under conventional laboratory conditions.

show abstract

“…While our study only examined speech perception, a similar approach could be taken to compare real and synthetic faces in other domains, such as emotions and looking behavior ( Miller et al, 2023 ).…”

Section: Discussionmentioning

confidence: 99%

Synthetic faces generated with the facial action coding system or deep neural networks improve speech-in-noise perception, but not as much as real faces

Yu,

Lado,

Zhang

et al. 2024

Front. Neurosci.

View full text Add to dashboard Cite

The prevalence of synthetic talking faces in both commercial and academic environments is increasing as the technology to generate them grows more powerful and available. While it has long been known that seeing the face of the talker improves human perception of speech-in-noise, recent studies have shown that synthetic talking faces generated by deep neural networks (DNNs) are also able to improve human perception of speech-in-noise. However, in previous studies the benefit provided by DNN synthetic faces was only about half that of real human talkers. We sought to determine whether synthetic talking faces generated by an alternative method would provide a greater perceptual benefit. The facial action coding system (FACS) is a comprehensive system for measuring visually discernible facial movements. Because the action units that comprise FACS are linked to specific muscle groups, synthetic talking faces generated by FACS might have greater verisimilitude than DNN synthetic faces which do not reference an explicit model of the facial musculature. We tested the ability of human observers to identity speech-in-noise accompanied by a blank screen; the real face of the talker; and synthetic talking faces generated either by DNN or FACS. We replicated previous findings of a large benefit for seeing the face of a real talker for speech-in-noise perception and a smaller benefit for DNN synthetic faces. FACS faces also improved perception, but only to the same degree as DNN faces. Analysis at the phoneme level showed that the performance of DNN and FACS faces was particularly poor for phonemes that involve interactions between the teeth and lips, such as /f/, /v/, and /th/. Inspection of single video frames revealed that the characteristic visual features for these phonemes were weak or absent in synthetic faces. Modeling the real vs. synthetic difference showed that increasing the realism of a few phonemes could substantially increase the overall perceptual benefit of synthetic faces.

show abstract

How do people respond to computer-generated versus human faces? A systematic review and meta-analyses

Cited by 11 publications

References 134 publications

The Effect on Speech-in-Noise Perception of Real Faces and Synthetic Faces Generated with either Deep Neural Networks or the Facial Action Coding System

The Effect on Speech-in-Noise Perception of Real Faces and Synthetic Faces Generated with either Deep Neural Networks or the Facial Action Coding System

Electrophysiological correlates of face and object perception: A comparative analysis of 2D laboratory and virtual reality conditions

Synthetic faces generated with the facial action coding system or deep neural networks improve speech-in-noise perception, but not as much as real faces

Contact Info

Product

Resources

About