Facial Expression Based Imagination Index and a Transfer Learning Approach to Detect Deception

Hasan, K. M. Azharul; Rahman, Wasifur; Gerstner, Luke; Sen, Taylan; Lee, Sangwu; Haut, Kurtis; Hoque, Mohammed Ehsan

doi:10.1109/acii.2019.8925473

Cited by 5 publications

(2 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Analysis based on the unimodal signal is not fully inclsuive of real-world characteristics and could lead to misleading findings (Braga and Mar-ques, 2004;Straßmann et al, 2016;Hasan et al, 2019c). That's why there exists vast amount prior research that utilize multimodal data to understand human communication behavior properly (Rahman et al, 2020;Zadeh et al, 2018a;Tsai et al, 2019;Samrose et al, 2019;Sen et al, 2018;Hasan et al, 2019b;Zadeh et al, 2018b;Hasan et al, 2019a). Petukhova et al (2017) discuss the design and evaluation of a Virtual Debate Coach (VDC) for training young politicians to improve their debate skills.…”

Section: Related Workmentioning

confidence: 99%

Hitting your MARQ: Multimodal ARgument Quality Assessment in Long Debate Video

Hasan¹,

Spann²,

Hasan³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

The combination of gestures, intonations, and textual content plays a key role in argument delivery. However, the current literature mostly considers textual content while assessing the quality of an argument, and is limited to datasets containing short sequences (18-48 words). In this paper, we study argument quality assessment in a multimodal context, and experiment on DBATES, a publicly available dataset of long debate videos. First, we propose a set of interpretable debate-centric features such as clarity, content variation, body movement cues, and pauses, inspired by theories of argumentation quality. Second, we design the Multimodal ARgument Quality assessor (MARQ) -a hierarchical neural network model that summarizes the multimodal signals on long sequences and enriches the multimodal embedding with debate-centric features. Our proposed MARQ model achieves an accuracy of 81.91% on the argument quality prediction task and outperforms established baseline models with an error rate reduction of 22.7%. Through ablation studies, we demonstrate the importance of multimodal cues in modeling argument quality.

show abstract

Section: Related Workmentioning

confidence: 99%

Hitting your MARQ: Multimodal ARgument Quality Assessment in Long Debate Video

Hasan¹,

Spann²,

Hasan³

et al. 2021

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…To address the scarcity of labeled high-stakes deception data for training models [14], prior re-arXiv:2102.03673v1 [cs.CV] 6 Feb 2021 search has focused on developing models that are robust to small numbers of samples [14,15]. While supervised transfer learning models have been developed to detect deception [16,17], to the best of our knowledge, no existing research has introduced unsupervised models to address the data scarcity problem of high-stakes deception detection.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Audio-Visual Subspace Alignment for High-Stakes Deception Detection

Mathur

Matarić²

2021

ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Automated systems that detect deception in high-stakes situations can enhance societal well-being across medical, social work, and legal domains. Existing models for detecting highstakes deception in videos have been supervised, but labeled datasets to train models can rarely be collected for most realworld applications. To address this problem, we propose the first multimodal unsupervised transfer learning approach that detects real-world, high-stakes deception in videos without using high-stakes labels. Our subspace-alignment (SA) approach adapts audio-visual representations of deception in lab-controlled low-stakes scenarios to detect deception in real-world, high-stakes situations. Our best unsupervised SA models outperform models without SA, outperform human ability, and perform comparably to a number of existing supervised models. Our research demonstrates the potential for introducing subspace-based transfer learning to model highstakes deception and other social behaviors in real-world contexts with a scarcity of labeled behavioral data.

show abstract