Exposing AI-generated videos with motion magnification

Fei, Jianwei; Xia, Zhihua; Yu, Peipeng; Xiao, Fengjun

doi:10.1007/s11042-020-09147-3

Cited by 21 publications

(4 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Chen et al [56] proposed an XceptionNet-ConvLSTM network with an added spatiotemporal attention mechanism to extract more robust temporal features and improve the generalizability of deepfake face detection. Fei et al [57] initially preprocessed the videos by applying Eulerian motion magni cation to amplify the facial motions, which considerably enhanced the distortion or icker in fake videos. The enhanced frames sequences were then used to train a joint CNN-LSTM network and distinguish between real and fake faces.…”

Section: Video Deepfake Detectionmentioning

confidence: 99%

Attention-based Multimodal learning framework for Generalized Audio- Visual Deepfake Detection

Masood,

Javed,

Irtaza

2023

Preprint

View full text Add to dashboard Cite

Deepfake media proliferated on the internet has major societal consequences for politicians, celebrities, and even common people. Recent advancements in deepfake videos include the creation of realistic talking faces and the usage of synthetic human voices. Numerous deepfake detection approaches have been proposed in response to the potential harm caused by deepfakes. However, the majority of deepfake detection methods process audio and video modality independently and have low identification accuracy. In this work, we propose an ensemble multimodal deepfake detection method that can identify both auditory and facial manipulations by exploiting correspondence between audio-visual modalities. The proposed framework comprises unimodal and cross-modal learning networks to exploit intra- and inter-modality inconsistencies introduced as a result of manipulation. The suggested multimodal approach employs an ensemble of deep convolutional neural-network based on an attention mechanism that extracts representative features and effectively determines if a video is fake or real. We evaluated the proposed approach on several benchmark multimodal deepfake datasets including FakeAVCeleb, DFDC-p, and DF-TIMIT. Experimental results demonstrate that an ensemble of deep learners based on unimodal and cross-modal network mechanisms exploit highly semantic information between audio and visual signals and outperforms independently trained audio and visual classifiers. Moreover, it can effectively identify different unseen types of deepfakes as well as robust under various post-processing attacks. The results confirm that our approach outperforms existing unimodal/multimodal classifiers for audio-visual manipulated video identification.

show abstract

Section: Video Deepfake Detectionmentioning

confidence: 99%

Attention-based Multimodal learning framework for Generalized Audio- Visual Deepfake Detection

Masood,

Javed,

Irtaza

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, Sarrafi et al [5] and Eitner et al [6] used point tracking on motion magnified recordings for the detection and modal parameter estimation of vibrations in different mechanical components. Fei et al [7] applied motion magnification to the detection of AI-generated videos and Alinovi et al…”

Section: Introductionmentioning

confidence: 99%

Lagrangian motion magnification with double sparse optical flow decomposition

Flotho,

Heiss,

Steidl

et al. 2023

Front. Appl. Math. Stat.

View full text Add to dashboard Cite

Microexpressions are fast and spatially small facial expressions that are difficult to detect. Therefore, motion magnification techniques, which aim at amplifying and hence revealing subtle motion in videos, appear useful for handling such expressions. There are basically two main approaches, namely, via Eulerian or Lagrangian techniques. While the first one magnifies motion implicitly by operating directly on image pixels, the Lagrangian approach uses optical flow (OF) techniques to extract and magnify pixel trajectories. In this study, we propose a novel approach for local Lagrangian motion magnification of facial micro-motions. Our contribution is 3-fold: first, we fine tune the recurrent all-pairs field transforms (RAFT) for OFs deep learning approach for faces by adding ground truth obtained from the variational dense inverse search (DIS) for the OF algorithm applied to the CASME II video set of facial micro expressions. This enables us to produce OFs of facial videos in an efficient and sufficiently accurate way. Second, since facial micro-motions are both local in space and time, we propose to approximate the OF field by sparse components both in space and time leading to a double sparse decomposition. Third, we use this decomposition to magnify micro-motions in specific areas of the face, where we introduce a new forward warping strategy using a triangular splitting of the image grid and barycentric interpolation of the RGB vectors at the corners of the transformed triangles. We demonstrate the feasibility of our approach by various examples.

show abstract

“…Although the detection of fake videos has received a lot of attention in recent years 6 , 7 , the research on the detection of misleading videos is still lacking. Since fake videos are synthetic videos in nature, the detection approaches are mainly based on the computer vision technology such as tampering detection 8 , copy-move forgery detection 9 and motion magnification detection 10 . As a result, they cannot be applied for the misleading videos directly.…”

Section: Introductionmentioning

confidence: 99%

A CNN-based misleading video detection model

Xiao

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Videos, especially short videos, have become an increasingly important source of information in these years. However, many videos spread on video sharing platforms are misleading, which have negative social impacts. Therefore, it is necessary to find methods to automatically identify misleading videos. In this paper, three categories of features (content features, uploader features and environment features) are proposed to construct a convolutional neural network (CNN) for misleading video detection. The experiment showed that all the three proposed categories of features play a vital role in detecting misleading videos. Our proposed approach that combines three categories of features achieved the best performance with the accuracy of 0.90 and the F1 score of 0.89. It also outperformed other baselines such as SVM, k-NN, decision tree and random forest models by more than 22%.

show abstract

Exposing AI-generated videos with motion magnification

Cited by 21 publications

References 35 publications

Attention-based Multimodal learning framework for Generalized Audio- Visual Deepfake Detection

Attention-based Multimodal learning framework for Generalized Audio- Visual Deepfake Detection

Lagrangian motion magnification with double sparse optical flow decomposition

A CNN-based misleading video detection model

Contact Info

Product

Resources

About