Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos

Abstract

Detecting manipulated images and videos is an important topic in digital media forensics. Most detection methods use binary classification to determine the probability of a query being manipulated. Another important topic is locating manipulated regions (i.e., performing segmentation), which are mostly created by three commonly used attacks: removal, copy-move, and splicing. We have designed a convolutional neural network that uses the multi-task learning approach to simultaneously detect manipulated images and videos and locate the manipulated regions for each query. Information gained by performing one task is shared with the other task and thereby enhance the performance of both tasks. A semi-supervised learning approach is used to improve the network's generability. The network includes an encoder and a Y-shaped decoder. Activation of the encoded features is used for the binary classification. The output of one branch of the decoder is used for segmenting the manipulated regions while that of the other branch is used for reconstructing the input, which helps improve overall performance. Experiments using the FaceForensics and Face-Forensics++ databases demonstrated the networks effectiveness against facial reenactment attacks and face swapping attacks as well as its ability to deal with the mismatch condition for previously seen attacks. Moreover, fine-tuning using just a small amount of data enables the network to deal with unseen attacks.

iCaps-Dfake: An Integrated Capsule-Based Model for Deepfake Image and Video Detection

“…They created two variants; a multilayer feedforward neural network (VA-MLP) and a logistic regression model (VA-LogReg). Another model, Multi-task, was created by [53] used a CNN to perform a multi-task learning problem by classifying fake videos and specifying manipulated areas. In FakeSpotter [54], Wang et al managed to overcome noise and distortions by monitoring the pattern of each layer's neuron activation of a face recognition network to seize the fine features that could help in detecting fakes.…”

“…Table 1 shows a summary of the most used datasets addressing this domain where the visual manipulations are precisely face replacements and only two of them included additional audio manipulations. In [35], it was proven that Celeb-DF and DFDC-P are very challenging datasets based on calculating the average AUC metric when compared to other available datasets using Two-Stream [49], MesoNet [50], HeadPose [39], FWA [31], VA [52], Xception [34], Multi-Task [53], Capsule [61] and DSP-FWA [32] detection methods. This have led us to focus our experiments on those two reliable benchmarks on which multiple trials were performed until the best results were achieved.…”

Generation and Detection of Media Clones

“…Accuracy (%) Replay Attack [69] 00.00 CGvsPhotos -Patches [26] 97.00 CGvsPhotos -Full size [26] 100.00 DeepFakes -Frame level [27] 95.93 DeepFakes -Video level [27] 99.23 FaceForensics -No compression [70] 99.37 FaceForensics -Light compression [70] 96.50 FaceForensics -Strong compression [70] 81.00 4.1 Detection of Computer Generated/Manipulated Faces [66], [67] To detect computer generated/manipulated images, we proposed a high-performance classifier [66] based on the capsule network architecture [68], which has fewer parameters than traditional CNNs. The proposed capsule module consists of three primary capsules and two output capsules, one for real and one for fake images.…”

“…We introduced a Y-shape autoencoder network (Fig. 5) that uses multi-task learning and semi-supervised learning approaches to simultaneously detect manipulated images/video frames and locate the manipulated regions [67]. Activation of the encoded features is used for classification.…”

Attentive Semantic Exploring for Manipulated Face Detection

“…Scholars try to tackle diverse types of fake face images with multifarious ideas in recent studies [21,22,23,24,25,26,27]. For instance, [21] proposes an auto-encoder-based model to detect manipulated face images. [23] puts forward an attention-based CNN to locate manipulation regions in fake images.…”