Gaze, Nod, and Smile for Success. A Deepfake-Based Study on Facial Expressiveness and Social Outcomes

Abstract

Artificial intelligence (AI)-generated media is used in entertainment, art, education, and marketing. AI-generated faces or facial expressions using deepfake-based technologies might also contribute to nonverbal behavior studies. As a proof of concept, in this research, we test the replicability of past results regarding the positive effects of facial expressiveness (e.g., gazing, nodding, and smiling) on social outcomes. Because the facial expressions when listening to someone (i.e., nonverbal immediacy) encompass several behaviors that are typically related (i.e., smiling while nodding), the effect of combinations of these behaviors is challenging to study. We thus test the effect of facial expressiveness (gazing, nodding, and smiling vs. none) on first impression judgements in a job interview setting using AI-generated experimental video material. We measure how competent, warm, and favorably independent observers (n = 823) perceive the targets (n = 159) showing AI-generated facial expressions. Our results replicate past research showing that the more targets are facially expressive, the more they are perceived favorably. Apart from supporting evidence of the importance of facial expressiveness for conveying a positive impression, we show the benefits of relying on AI-generated experimental video material for the study of nonverbal behavior.