Localization of Deep Video Inpainting Based on Spatiotemporal Convolution and Refinement Network

Abstract

Deep learning-based video inpainting can fill the missing or undesired regions with spatial-temporal consistent contents without obvious visually distortion. Although the original purpose of deep inpainting is to repair flawed videos, it can also be adopted for malicious purposes, e.g., removal of specific objects. Therefore, automatically locating the inpainted regions is a challenging task in video forensics. This paper proposes a new forensic refinement framework to localize the deep inpainted regions by considering the spatial-temporal viewpoint. Firstly, we design a spatiotemporal convolution to suppress redundancy for highlighting deep inpainting traces. Then, a detection module is constructed with four concatenated ResNet blocks, and two upsampling layers to achieve a rough location map. Finally, a modified U-net based refinement module is developed for the pixel-wise localization map. Deep inpaiting video datasets created by the state-of-the-art deep inpainting method, have been evaluated, and extensive experimental results clearly demonstrate the efficacy of the proposed approach.

An Efficient Video Inpainting Approach Using Deep Belief Network

“…Now, the hybridization of 2 Meta heuristic approaches such as CS and MVO algorithms named CSMVO is employed for optimizing the patch matching and RNN model. Ding et al [13] proposed a novel forensic refinement architecture for localizing the deep in painted region with the consideration of spatial temporal viewpoints. Initially, designed spatiotemporal convolutions for suppressing redundancy to highlight deep inpainting traces.…”

Video matting tampering detection based on time and space domain traces

“…With the development of deep learning, deep learning models can make better use of the temporal information in video sequences by continuously analysing and modelling the video frames, to more accurately detect tampering behaviours in videos, including object motion, dynamic changes, etc. However, there are only a few methods that try to deal with video matting tampering detection [7,8] and are underutilised for tampering edge traces. On the other hand, several image forensics methods can locate tampered video regions frame by frame [9], but they do not take advantage of the temporal correlation between video frames and perform poorly.…”