Deception Detection in Videos

Abstract

We present a system for covert automated deception detection using information available in a video. We study the importance of different modalities like vision, audio and text for this task. On the vision side, our system uses classifiers trained on low level video features which predict human micro-expressions. We show that predictions of high-level micro-expressions can be used as features for deception prediction. Surprisingly, IDT (Improved Dense Trajectory) features which have been widely used for action recognition, are also very good at predicting deception in videos. We fuse the score of classifiers trained on IDT features and high-level micro-expressions to improve performance. MFCC (Mel-frequency Cepstral Coefficients) features from the audio domain also provide a significant boost in performance, while information from transcripts is not very beneficial for our system. Using various classifiers, our automated system obtains an AUC of 0.877 (10-fold cross-validation) when evaluated on subjects which were not part of the training set. Even though state-of-the-art methods use human annotations of micro-expressions for deception detection, our fully automated approach outperforms them by 5%. When combined with human annotations of micro-expressions, our AUC improves to 0.922. We also present results of a user-study to analyze how well do average humans perform on this task, what modalities they use for deception detection and how they perform if only one modality is accessible.

Affect-Aware Machine Learning Models for Deception Detection

“…Facial affect contributed towards the best multimodal approach, which obtained an AUC of 91% and accuracy of 84% through adaptive boosting (AdaBoost) across facial affect, visual, and vocal modalities. The 91% AUC achieved by our approach was higher than the AUC of the best-performing automated approach on this dataset (88% AUC) that did not use facial affect, but also used an SVM with interpretable visual, vocal, and verbal features (Wu et al 2018). These results demonstrate the discriminative power of facial affect as a feature set in multimodal machine learning models for automated deception detection.…”

Deception Detection in Group Video Conversations using Dynamic Interaction Networks

“…Computer vision baselines. We compare our method with five computer vision baselines with the same experimental setup as our method Baltrusaitis et al 2018;Demyanov et al 2015;Wu et al 2018). These methods used features extracted from the video, including facial emotion, head and eye movement, facial action units, and time-aggregated features as described below.…”

“…(Baltrusaitis et al 2018) computes eye movements from the estimated eye ball positions, and uses the movement distributions over time as features. (Wu et al 2018) extracts the individual dense trajectory features from videos, MFCC features from audio, micro-expression features and text features from transcripts, and uses an ensemble method called late fusion to come up with a joint prediction. Since our dataset doesn't have transcripts and annotated micro-expressions, we remove the text features and replace micro-expressions by FAU features (Demyanov et al 2015).…”

“…While there has been significant research in identifying individual deceivers in real-world face-to-face interactions (Wu et al 2018;Ding et al 2019;Gogate, Adeel, and Hussain 2017), little is known about how groups of deceivers work together in the online setting. Current deception research is largely limited to analysing the audio-visual behavior of a single deceiver using voice signatures, gestures, facial expressions, and body language.…”

Explainable Verbal Deception Detection using Transformers

“…However, most approaches in deception detection use only traditional (ensemble) machine learning methods, such as logistic regression, support vector machines, random forests, or naive bayes classifiers to predict veracity (Wu et al, 2018;Banerjee et al, 2015;Fornaciari and Poesio, 2013). Only few studies have explored deep learning methods based on transformer-networks (Fornaciari et al, 2021;Kao et al, 2020;Kennedy et al, 2019), which show increased prediction performances over the traditional approaches, but do not focus on interpreting the linguistic properties of the texts.…”