Attention-Based Fusion of Ultrashort Voice Utterances and Depth Videos for Multimodal Person Identification

Moufidi, Abderrazzaq; Rousseau, David; Rasti, Pejman

doi:10.3390/s23135890

Cited by 2 publications

(1 citation statement)

References 35 publications

(104 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Several modal fusion techniques have been well developed in various domains in recent years. For example, multimodal emotion recognition [28][29][30], multimodal interaction [31], and lip-sync fusion [32][33][34][35] have seen significant advancements. In 2023, Qin et al [36] provided a comprehensive review of identification techniques and applications in unimodal identity recognition.…”

Section: Audio-visual Person Verificationmentioning

confidence: 99%

Audio–Visual Fusion Based on Interactive Attention for Person Verification

Jing,

He,

Song

et al. 2023

Sensors

View full text Add to dashboard Cite

With the rapid development of multimedia technology, personnel verification systems have become increasingly important in the security field and identity verification. However, unimodal verification systems have performance bottlenecks in complex scenarios, thus triggering the need for multimodal feature fusion methods. The main problem with audio–visual multimodal feature fusion is how to effectively integrate information from different modalities to improve the accuracy and robustness of the system for individual identity. In this paper, we focus on how to improve multimodal person verification systems and how to combine audio and visual features. In this study, we use pretrained models to extract the embeddings from each modality and then perform fusion model experiments based on these embeddings. The baseline approach in this paper involves taking the fusion feature and passing it through a fully connected (FC) layer. Building upon this baseline, we propose three fusion models based on attentional mechanisms: attention, gated, and inter–attention. These fusion models are trained on the VoxCeleb1 development set and tested on the evaluation sets of the VoxCeleb1, NIST SRE19, and CNC-AV datasets. On the VoxCeleb1 dataset, the best system performance achieved in this study was an equal error rate (EER) of 0.23% and a detection cost function (minDCF) of 0.011. On the evaluation set of NIST SRE19, the EER was 2.60% and the minDCF was 0.283. On the evaluation set of the CNC-AV set, the EER was 11.30% and the minDCF was 0.443. These experimental results strongly demonstrate that the proposed fusion method can significantly improve the performance of multimodal character verification systems.

show abstract

Section: Audio-visual Person Verificationmentioning

confidence: 99%

Audio–Visual Fusion Based on Interactive Attention for Person Verification

Jing,

He,

Song

et al. 2023

Sensors

View full text Add to dashboard Cite

show abstract

Toward comprehensive short utterances manipulations detection in videos

Moufidi,

Rousseau,

Rasti

2024

Multimed Tools Appl

View full text Add to dashboard Cite

In a landscape increasingly populated by convincing yet deceptive multimedia content generated through generative adversarial networks, there exists a significant challenge for both human interpretation and machine learning algorithms. This study introduces a shallow learning technique specifically tailored for analyzing visual and auditory components in videos, targeting the lower face region. Our method is optimized for ultra-short video segments (200-600 ms) and employs wavelet scattering transforms for audio and discrete cosine transforms for video. Unlike many approaches, our method excels at these short durations and scales efficiently to longer segments. Experimental results demonstrate high accuracy, achieving 96.83% for 600 ms audio segments and 99.87% for whole video sequences on the FakeAVCeleb and DeepfakeTIMIT datasets. This approach is computationally efficient, making it suitable for real-world applications with constrained resources. The paper also explores the unique challenges of detecting deepfakes in ultra-short sequences and proposes a targeted evaluation strategy for these conditions.

show abstract

Attention-Based Fusion of Ultrashort Voice Utterances and Depth Videos for Multimodal Person Identification

Cited by 2 publications

References 35 publications

Audio–Visual Fusion Based on Interactive Attention for Person Verification

Audio–Visual Fusion Based on Interactive Attention for Person Verification

Toward comprehensive short utterances manipulations detection in videos

Contact Info

Product

Resources

About