Video Codec Forensics Based on Convolutional Neural Networks

Abstract

The recent development of multimedia has made video editing accessible to everyone. Unfortunately, forensic analysis tools capable of detecting traces left by video processing operations in a blind fashion are still at their beginnings. One of the reasons is that videos are customary stored and distributed in a compressed format, and codec-related traces tends to mask previous processing operations. In this paper, we propose to capture video codec traces through convolutional neural networks (CNNs) and exploit them as an asset. Specifically, we train two CNNs to extract information about the used video codec and coding quality, respectively. Building upon these CNNs, we propose a system to detect and localize temporal splicing for video sequences generated from the concatenation of different video segments, which are characterized by inconsistent coding schemes and / or parameters (e.g., video compilations from different sources or broadcasting channels). The proposed solution is validated using videos at different resolutions (i.e., CIF, 4CIF, PAL and 720p) encoded with four common codecs (i.e., MPEG2, MPEG4, H264 and H265) at different qualities (i.e., different constant and variable bitrates, as well as constant quantization parameters).

FOCAL: A Forgery Localization Framework Based on Video Coding Self-Consistency

“…In [44] the authors propose a recursive autoencoder (implemented with the LSTM architecture, to exploit temporal dependencies) to learn a feature representation of pristine videos and detect forgeries as outliers of the learned model. In [12] two CNNs are independently trained to extract codec and quality related features with the purpose of detecting temporal inconsistencies, showing that the combination of heterogeneous detectors enhances the overall performance. Some studies have also been addressed to expose the newly appeared threat of AIgenerated highly-realistic forgeries, also known as DeepFakes.…”

“…ROC curves for frame-wise temporal splicing localization. Comparison of codec-related, quality-related and combined features, for the proposed method and the baseline [12]. inconsistencies in the forged sequences: temporally-spliced videos have the first 100 frames encoded differently from the subsequent ones; spatially-spliced videos have a CIF window in the middle of the frame encoded differently from the rest.…”

“…Figure 6 shows the ROC curves obtained for temporal splicing localization with the three different set of features. For each case, a direct comparison with the baseline work in [12] is provided. All three descriptors outperform the baseline ones, with an AUC peaking at 0.984 for the concatenated features.…”

“…Experimental validation takes into account different forgery setups, such as temporal and spatial splicing in controlled and uncontrolled environments. Results show that the proposed solution is able to improve the performance of [12], thanks to the newly adopted network architecture, and to obtain convincing results in the detection of local forgeries as well, with an area under the curve (AUC) of the receiver operating characteristic (ROC) curve of 0.94. The rest of the paper is organized as follows.…”

“…Extending the preliminary work in [12], the current paper proposes a FOrgery loCALizer (FOCAL) that checks the selfconsistency of multiple and independent forensic traces related to video coding (Figure 1). Differently from the previous work where forgeries consisted in concatenating video sequences from different sources (temporal splicing), this new approach is also able to precisely localize an altered region within a single frame (spatial splicing) as well as along time dimension.…”