Dual-modality Talking-metrics: 3D Visual-Audio Integrated Behaviometric Cues from Speakers

Abstract

Face-based behaviometrics focus on dynamic biological signatures generated from face behaviors, which are informative and subject-specific for identity recognition. Most existing face behaviometrics rely on 2D visual features and thus are sensitive to pose or intensity variations. This paper presents a dual-modality behaviometrics algorithm (talking-metrics) that integrates 3D video and audio cues from a human face speaking a passphrase. Static and dynamic 3D face features are extracted algorithmically and audio features are transformed through a few learning models. We concatenate the top 18 discriminative 3D visual-audio features to represent the bi-modality and utilize an linear discrimant analysis (LDA) classifier for identity recognition. The experiments were conducted on a new publicly released dataset (S3DFM). Both qualitative feature distributions and quantitative comparison results show the feasibility of the proposed pipeline and the superiority over using each modality independently. A 98.5% cross-validation recognition rate over 60 subjects and 10 trials was achieved. An anti-spoofing test also demonstrates the robustness of the proposed method.

3D Lip Event Detection via Interframe Motion Divergence at Multiple Temporal Resolutions

“…The proposed pipeline was verified on a publicly available dynamic face dataset -Speech-driven 3D Facial Motion Dataset (S3DFM) [1,22]. The dataset has multimodality data from 77 subjects covering more than 20 nationalities.…”

“…The proposed algorithms were verified on a 3D speaking face dataset (S3DFM [20,22]) 1 and has good detection performance over 100 sequences with 200 events.…”

3D Visual passcode: Speech-driven 3D facial dynamics for behaviometrics

“…For each trial, we captured a pair of 2D intensity sequences using the 3D video sensor and a synchronized audio sequence using a microphone. The synchronized audio was not used in the validation algorithm here, but it was used in another 3D video-audio recognition related research [44].…”