Face-Focused Cross-Stream Network for Deception Detection in Videos

Abstract

Automated deception detection (ADD) from real-life videos is a challenging task. It specifically needs to address two problems: (1) Both face and body contain useful cues regarding whether a subject is deceptive. How to effectively fuse the two is thus key to the effectiveness of an ADD model. (2) Real-life deceptive samples are hard to collect; learning with limited training data thus challenges most deep learning based ADD models. In this work, both problems are addressed. Specifically, for face-body multimodal learning, a novel face-focused cross-stream network (FFCSN) is proposed. It differs significantly from the popular two-stream networks in that: (a) face detection is added into the spatial stream to capture the facial expressions explicitly, and (b) correlation learning is performed across the spatial and temporal streams for joint deep feature learning across both face and body. To address the training data scarcity problem, our FFCSN model is trained with both meta learning and adversarial learning. Extensive experiments show that our FFCSN model achieves state-of-the-art results. Further, the proposed FFCSN model as well as its robust training strategy are shown to be generally applicable to other human-centric video analysis tasks such as emotion recognition from user-generated videos.

Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection

“…Our unsupervised visual SA model performed comparably to existing fully-supervised, automated approaches that used the same dataset (ACC 75% [11], 77% [13], 79% [12]; AUC 70% [10]). While some prior fully-supervised, automated approaches [21,15,20] outperformed our unsupervised SA, our findings support the potential for introducing unsupervised SA to address the data scarcity problem of modeling high-stakes deception. Our results support our hypothesis that audio-visual representations of low-stakes deception in lab-controlled situations can be leveraged by SA to detect high-stakes deception in real-world situations.…”

Machine Learning-Based Lie Detector Applied to a Novel Annotated Game Dataset

“…In addition, Soldner et al [32] introduced dialogue features, consisting of interaction cues. Other multi-modal approaches combined the previously mention verbal and non-verbal features together with micro-expressions [3][4][5], thermal imaging [33], or spatio-temporal features extracted from 3D CNNs [34,35].…”

“…In the last decade, there has been a growing interest in the use of facial images to perform lie detection, often based on micro-expressions [3][4][5]13,15] or facial action units [14], achieving the current state-of-the-art accuracy. Table 1 below shows an overview of the major related works outlined in this section.…”

“…It is considered hard for humans to detect when someone is lying. Ekman [1] highlights five reasons to explain why it is so difficult for us: (1) during most of human history, there were smaller societies in which liars would have had more chances of being caught with worse consequences than nowadays; (2) children are not taught how to detect lies since even their parents want to hide some things from them; (3) people prefer to trust in what they are told; (4) people prefer not to know the real truth; and (5) people are taught to be polite and not steal information that is not given. However, it has been argued that it is possible for someone to learn how to detect lies in another person given sufficient feedback (e.g., that 50% of the time, that person is lying) and focusing on micro-expressions [1,2].…”

“…Building from the above, the detection of deceptive behavior using facial analysis has been proved feasible using macro-and, especially, micro-expressions [3][4][5]. However, micro-expressions are difficult to capture at standard frame rates and, given that humans can learn how to spot them to perform lie detection, the same training might be used by liars to learn how to hide them.…”