Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations

Matern, Falko; Rieß, Christian; Stamminger, Marc

doi:10.1109/wacvw.2019.00020

Cited by 521 publications

(273 citation statements)

References 21 publications

Supporting

Mentioning

270

Contrasting

Unclassified

Order By: Relevance

“…As shown in Fig.1, the current state of the art in GANs is far from perfection, and often generated images exhibit strong visual artifacts that can be exploited for forensic use. For example, to detect fake faces, [15] exploits visual features regarding eyes, teeth and facial contours. Tellingly, the authors observe that in GAN-generated images the color of left and right eye are often inconsistent.…”

Section: Related Workmentioning

confidence: 99%

Do GANs Leave Artificial Fingerprints?

Marra

Gragnaniello

Verdoliva

et al. 2019

2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)

223

171

View full text Add to dashboard Cite

In the last few years, generative adversarial networks (GAN) have shown tremendous potential for a number of applications in computer vision and related fields. With the current pace of progress, it is a sure bet they will soon be able to generate high-quality images and videos, virtually indistinguishable from real ones. Unfortunately, realistic GAN-generated images pose serious threats to security, to begin with a possible flood of fake multimedia, and multimedia forensic countermeasures are in urgent need. In this work, we show that each GAN leaves its specific fingerprint in the images it generates, just like real-world cameras mark acquired images with traces of their photo-response non-uniformity pattern. Source identification experiments with several popular GANs show such fingerprints to represent a precious asset for forensic analyses.

show abstract

Section: Related Workmentioning

confidence: 99%

Do GANs Leave Artificial Fingerprints?

Marra

Gragnaniello

Verdoliva

et al. 2019

2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)

223

171

View full text Add to dashboard Cite

show abstract

“…For instance, Li et al [36] propose a Deep Neural Network (DNN) to detect fake videos based on artifacts observed during the face warping step of the generation algorithms. Similarly, Yang et al [61] look at inconsistencies in the head poses in the synthesized videos and Matern et al [38] capture artifacts in the eyes, teeth and facial contours of the generated faces. Prior works have also experimented with a variety of network architectures.…”

Section: Unimodal Deepfake Detection Methodsmentioning

confidence: 99%

“…and algorithms [9,26,36,38,42,43,49,50,57,61,64] for deepfake detection. DeepFake detection methods classify an input video or image as "real" or "fake".…”

mentioning

confidence: 99%

Emotions Don't Lie

Mittal

Bhattacharya

Chandra

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

130

View full text Add to dashboard Cite

We present a learning-based method for detecting real and fake deepfake multimedia content. To maximize information for learning, we extract and analyze the similarity between the two audio and visual modalities from within the same video. Additionally, we extract and compare affective cues corresponding to perceived emotion from the two modalities within a video to infer whether the input video is "real" or "fake". We propose a deep learning network, inspired by the Siamese network architecture and the triplet loss. To validate our model, we report the AUC metric on two large-scale deepfake detection datasets, DeepFake-TIMIT Dataset and DFDC. We compare our approach with several SOTA deepfake detection methods and report per-video AUC of 84.4% on the DFDC and 96.6% on the DF-TIMIT datasets, respectively. To the best of our knowledge, ours is the first approach that simultaneously exploits audio and video modalities and also perceived emotions from the two modalities for deepfake detection. CCS CONCEPTS • Computing methodologies → Visual content-based indexing and retrieval; Neural networks.

show abstract

“…In the Multi-Task Multi-Classifier (MT-MC) variant, a separate binary detector is used, characterized by its own loss BL(Xn, Φ) = x i ∈Xn δY=Gd(xi) + δY =R(1 − d(xi)) (7) where d(x i ) is the detector output and Y is the binary label of interest. The network is trained by using together this loss and the usual iCARL loss, in a classical multi-task learning fashion, with aggregated loss:…”

Section: Proposed Methodsmentioning

confidence: 99%

“…Several methods have been already proposed in the literature to detect whether an image is GAN-generated or not. Some of them exploit specific facial artifacts, like asymmetries in the colour of the eyes, or artifacts arising from an imprecise estimation of the underlying geometry, especially on areas around the nose, the border of the face, and the eyebrows [7]. Color information is instead exploited in [8], [9].…”

Section: Introductionmentioning

confidence: 99%

Incremental learning for the detection and classification of GAN-generated images

Marra

Saltori

Boato

et al. 2019

2019 IEEE International Workshop on Information Forensics and Security (WIFS)

View full text Add to dashboard Cite

Current developments in computer vision and deep learning allow to automatically generate hyper-realistic images, hardly distinguishable from real ones. In particular, human face generation achieved a stunning level of realism, opening new opportunities for the creative industry but, at the same time, new scary scenarios where such content can be maliciously misused. Therefore, it is essential to develop innovative methodologies to automatically tell apart real from computer generated multimedia, possibly able to follow the evolution and continuous improvement of data in terms of quality and realism. In the last few years, several deep learning-based solutions have been proposed for this problem, mostly based on Convolutional Neural Networks (CNNs). Although results are good in controlled conditions, it is not clear how such proposals can adapt to realworld scenarios, where learning needs to continuously evolve as new types of generated data appear. In this work, we tackle this problem by proposing an approach based on incremental learning for the detection and classification of GAN-generated images. Experiments on a dataset comprising images generated by several GAN-based architectures show that the proposed method is able to correctly perform discrimination when new GANs are presented to the network.

show abstract

Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations

Cited by 521 publications

References 21 publications

Do GANs Leave Artificial Fingerprints?

Do GANs Leave Artificial Fingerprints?

Emotions Don't Lie

Incremental learning for the detection and classification of GAN-generated images

Contact Info

Product

Resources

About