EEG-Based Analysis of the Impact of Familiarity in the Perception of Deepfake Videos

Abstract

We investigate the brain's subliminal response to fake portrait videos using electroencephalography, with a special emphasis on the viewer's familiarity with the depicted individuals.Deepfake videos are increasingly becoming popular but, while they are entertaining, they can also pose a threat to society. These face-swapped videos merge physiognomy and behaviour of two different individuals, both strong cues used for recognizing a person. We show that this mismatch elicits different brain responses depending on the viewer's familiarity with the merged individuals.Using EEG, we classify perceptual differences of familiar and unfamiliar people versus their face-swapped counterparts. Our results show that it is possible to discriminate fake videos from genuine ones when at least one face-swapped actor is known to the observer. Furthermore, we indicate a correlation of classification accuracy with level of personal engagement between participant and actor, as well as with the participant's familiarity with the used dataset.

Embracing deepfakes and AI‐generated images in neuroscience research

“…The ultimate test might be whether real and AI‐generated faces elicit similar neural responses, given human expertise in face perception (Haxby et al, 2000). So far, research has shown that face‐swap deepfakes elicit similar neural responses to real videos, unless we know one of the people being portrayed (Tauscher et al, 2021). Another study revealed that while neural responses were largely similar, consciously undetectable fake faces can still elicit slightly different neural responses compared with photos (Moshel et al, 2022).…”

Real and Deepfake Face Recognition: An EEG Study on Cognitive and Emotive Implications

“…Recently, studies on electroencephalographic (EEG) correlates investigated viewers’ ability to distinguish familiar and unfamiliar people from their face-swapped counterparts. Results showed that it is possible to discriminate fake videos from genuine ones when at least one face-swapped actor is known to the observer [ 17 ].…”

Electroencephalographic Correlates in Synthetic and Real Emotional Face Stimulation

“…Results showed that it is possible to discriminate fake videos from genuine ones when at least one face-swapped actor is known to the observer. [7] Investigation of brain reactions to facial expressions is becoming a widespread research area, which aims at better understanding emotional processing and cognitive mechanisms. Even though traditional models suggest that facial identity and expression are processed in distinctive brain areas, the current findings highlight that emotion processing can have a strong influence on facial recognition and memory mechanisms [8].…”