FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset

Abstract

While the significant advancements have made in the generation of deepfakes using deep learning technologies, its misuse is a well-known issue now. Deepfakes can cause severe security and privacy issues as they can be used to impersonate a person's identity in a video by replacing his/her face with another person's face. Recently, a new problem of generating synthesized human voice of a person is emerging, where AI-based deep learning models can synthesize any person's voice requiring just a few seconds of audio. With the emerging threat of impersonation attacks using deepfake audios and videos, a new generation of deepfake detectors is needed to focus on both video and audio collectively. A large amount of good quality dataset is typically required to capture the real-world scenarios to develop a competent deepfake detector. Existing deepfake datasets either contain deepfake videos or audios, which are racially biased as well. Hence, there is a crucial need for creating a good video as well as audio deepfake dataset, which can be used to detect audio and video deepfake simultaneously. To fill this gap, we propose a novel Audio-Video Deepfake dataset (FakeAVCeleb) that contains not only deepfake videos but also respective synthesized lip-synced fake audios. We generate this dataset using the current most popular deepfake generation methods. We selected real YouTube videos of celebrities with four racial backgrounds (Caucasian, Black, East Asian, and South Asian) to develop a more realistic multimodal dataset that addresses racial bias, and further help develop multimodal deepfake detectors. We performed several experiments using state-of-the-art detection methods to evaluate our deepfake dataset and demonstrate the challenges and usefulness of our multimodal Audio-Video deepfake dataset.Preprint. Under review.

Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization

“…The number of faces in each frame is more than one. Recently, FakeAVCeleb [31] was released focusing on both face-swap and face-reenactment methods with manipulated audio and video. ForgeryNet [23] is the latest contribution to the growing list of deepfake detection datasets.…”

Deepfake Detection for Facial Images with Facemasks

“…Inspired by the emergence of DeepFakes algorithm to the public, various methods, i.e., FaceSwap [26], NeuralTextures [7], Face2Face [24], and FSGAN [13], have been proposed to synthesize hyper-realistic deepfake images that are unrecognizable to human eyes. These methods allowed to generate numerous deepfake datasets [8,12,16] for public usage in the research community. Furthermore, Wav2Lips [15] has shown a lip-synchronization network, generating lip-syncing arbitrary talking face videos with arbitrary speech.…”

“…In this work, we used FaceForensics++ [16] C40, a compressed version of original FaceForensics++, and FakeAVCeleb [8] to train each model and assess the models on each dataset. The number of fake/real images used in each dataset is provided in Table 2.…”

“…In particular, we used low-quality images, which are compressed C40 version of the dataset, to consider the realistic setting for manipulated videos and to provide certain levels of difficulty for the performance assessment. • FakeAVCeleb [8] is an Audio-Video Multimodal Deepfake Detection dataset that contains both video and audio deepfakes with accurate lip-sync. FakeAVCeleb dataset contains three different types of video deepfakes, FaceSwap, DeepFaceLab, and FSGAN.…”

A Novel Smart Deepfake Video Detection System

“…Overdub, iSpeech, and VoiceApp are instances of voice cloning open-access platforms that can generate synthesized deepfake sounds that nearly resemble the target human's speech [3]. The work of [4] is an example of these manipulation methods, which involves the creation of highly realistic deepfake videos with a precise lip-sync using a group of AI technologies; FaceSwap, FaceSwap GAN, DeepFaceLab, SV2TTS [5], and Wav2Lip [6].…”