Appearance Matters, So Does Audio: Revealing the Hidden Face via Cross-Modality Transfer

Kong, Chenqi; Chen, Baoliang; Yang, Wenhan; Li, Haoliang; Chen, Peilin; Wang, Shiqi

doi:10.1109/tcsvt.2021.3057457

Cited by 25 publications

(4 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Plus, Kong et al (2021) suggested a cross‐modality approach that uses visual and auditory information to discover the hidden face behind the deepfake material. Every recovered audio segment is matched with all false faces retrieved from the related deepfake video, and each fake face is coupled with one matching ground truth face.…”

Section: Deep Fake Detection Mechanismsmentioning

confidence: 99%

Deepfake detection using deep learning methods: A systematic and comprehensive review

Heidari,

Jafari Navimipour,

Dag

et al. 2023

WIREs Data Min & Knowl

View full text Add to dashboard Cite

Deep Learning (DL) has been effectively utilized in various complicated challenges in healthcare, industry, and academia for various purposes, including thyroid diagnosis, lung nodule recognition, computer vision, large data analytics, and human‐level control. Nevertheless, developments in digital technology have been used to produce software that poses a threat to democracy, national security, and confidentiality. Deepfake is one of those DL‐powered apps that has lately surfaced. So, deepfake systems can create fake images primarily by replacement of scenes or images, movies, and sounds that humans cannot tell apart from real ones. Various technologies have brought the capacity to change a synthetic speech, image, or video to our fingers. Furthermore, video and image frauds are now so convincing that it is hard to distinguish between false and authentic content with the naked eye. It might result in various issues and ranging from deceiving public opinion to using doctored evidence in a court. For such considerations, it is critical to have technologies that can assist us in discerning reality. This study gives a complete assessment of the literature on deepfake detection strategies using DL‐based algorithms. We categorize deepfake detection methods in this work based on their applications, which include video detection, image detection, audio detection, and hybrid multimedia detection. The objective of this paper is to give the reader a better knowledge of (1) how deepfakes are generated and identified, (2) the latest developments and breakthroughs in this realm, (3) weaknesses of existing security methods, and (4) areas requiring more investigation and consideration. The results suggest that the Conventional Neural Networks (CNN) methodology is the most often employed DL method in publications. According to research, the majority of the articles are on the subject of video deepfake detection. The majority of the articles focused on enhancing only one parameter, with the accuracy parameter receiving the most attention.This article is categorized under: Technologies > Machine Learning Algorithmic Development > Multimedia Application Areas > Science and Technology

show abstract

Section: Deep Fake Detection Mechanismsmentioning

confidence: 99%

Deepfake detection using deep learning methods: A systematic and comprehensive review

Heidari,

Jafari Navimipour,

Dag

et al. 2023

WIREs Data Min & Knowl

View full text Add to dashboard Cite

show abstract

“…Dai et al [63] propose attentional local contrastive learning to capture local forgery information. Recent works [64]- [66] propose to learn consistency across different modalities, which boosts detectors to capture abundant forgery clues. However, finegrained features for general face forgery detection have been largely underexplored.…”

Section: B Face Forgery Detection Via Representation Learningmentioning

confidence: 99%

Dual Attention Network Approaches to Face Forgery Video Detection

Luo

Chen

2022

IEEE Access

View full text Add to dashboard Cite

Forged videos are commonly spread online. Most have malicious content and cause serious information security problems. The most critical issue in deepfake detection is the identification of traces of tampering in fake videos. This study designs a Dual Attention Forgery Detection Network (DAFDN), which embeds a spatial reduction attention block (SRAB) and a forgery feature attention module (FFAM) to the backbone network. DAFDN embeds the two proposed attention mechanisms and enables the convolution neural network to extract peculiar traces left by images' warping. This study uses two benchmark datasets, DFDC and FaceForensics++, to compare the performance of the proposed DAFDN with other methods. The results show that the proposed DAFDN mechanism achieves AUC scores of 0.911 and 0.945 in the datasets DFDC and FaceForensics++, respectively. These results are better than those of previously developed methods, such as XceptionNet and EfficientNet-related methods.

show abstract

“…This question has recently aroused much research interest [2,17,20,26,27,36,38,39,41,45], which we refer to as voice-face association learning (VFAL). This emerging research field has several promising applications, such as audio-visual speaker recognition [35], criminal profiling, speaker tracking [26], and deepfake video detection [22].…”

Section: Introductionmentioning

confidence: 99%

Self-Lifting: A Novel Framework for Unsupervised Voice-Face Association Learning

Chen

Zhang

Liu

et al. 2022

Proceedings of the 2022 International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

Voice-face association learning (VFAL) aims to tap into the potential connections between voices and faces. Most studies currently address this problem in a supervised manner, which cannot exploit the wealth of unlabeled video data. To solve this problem, we propose an unsupervised learning framework: Self-Lifting (SL), which can use unlabeled video data for learning. This framework includes two iterative steps of "clustering" and "metric learning". In the first step, unlabeled video data is mapped into the feature space by a coarse model. Then unsupervised clustering is leveraged to allocate pseudo-label to each video. In the second step, the pseudo-label is used as supervisory information to guide the metric learning process, which produces the refined model. These two steps are performed alternately to lift the model's performance. Experiments show that our framework can effectively use unlabeled video data for learning. On the VoxCeleb dataset, our approach achieves SOTA results among the unsupervised methods and has competitive performance compared with the supervised competitors. Our code is released on Github 1 . CCS CONCEPTS• Information systems → Information retrieval; • General and reference → General conference proceedings; • Computing methodologies → Computer vision tasks.

show abstract

Appearance Matters, So Does Audio: Revealing the Hidden Face via Cross-Modality Transfer

Cited by 25 publications

References 65 publications

Deepfake detection using deep learning methods: A systematic and comprehensive review

Deepfake detection using deep learning methods: A systematic and comprehensive review

Dual Attention Network Approaches to Face Forgery Video Detection

Self-Lifting: A Novel Framework for Unsupervised Voice-Face Association Learning

Contact Info

Product

Resources

About