Image Provenance Analysis at Scale

Abstract

Prior art has shown it is possible to estimate, through image processing and computer vision techniques, the types and parameters of transformations that have been applied to the content of individual images to obtain new images. Given a large corpus of images and a query image, an interesting further step is to retrieve the set of original images whose content is present in the query image, as well as the detailed sequences of transformations that yield the query image given the original images. This is a problem that recently has received the name of image provenance analysis. In these times of public media manipulation (e.g., fake news and meme sharing), obtaining the history of image transformations is relevant for fact checking and authorship verification, among many other applications. This article presents an end-to-end processing pipeline for image provenance analysis, which works at real-world scale. It employs a cutting-edge image filtering solution that is custom-tailored for the problem at hand, as well as novel techniques for obtaining the provenance graph that expresses how the images, as nodes, are ancestrally connected. A comprehensive set of experiments for each stage of the pipeline is provided, comparing the proposed solution with state-of-the-art results, employing previously published datasets. In addition, this work introduces a new dataset of real-world provenance cases from the social media site Reddit, along with baseline results.

Beyond Pixels: Image Provenance Analysis Leveraging Metadata

“…In order to overcome the direction limitation and propose a scalable approach, a more complete end-to-end pipeline for image provenance analysis was described in [52]. That method for graph construction first builds dissimilarity matrices based on local image features, and then employs hierarchical clustering to group nodes and draw edges within the final provenance graph.…”

“…As stated in Section 1, relying solely on image content can lead to noisy edge inference. This is especially true for directed edges, which have been shown to be more difficult to derive than undirected edges [11,52]. An option for addressing this is the use of metadata related to the images.…”

“…Image provenance analysis algorithms aim at constructing a provenance graph with related images, given a query image. The provenance graph [52] is a Directed Acyclic Graph (DAG) where each node corresponds to an image in the set of related images and the edges stand for the relationship of sharing duplicate content. The direction of an edge denotes the direction of the flow of content between each pair of images and the overall provenance case.…”

“…Due to the vast range of possible versions of a single original image, the metrics for quantifying the similarity between pairs of images can be noisy. Relying solely upon visual cues to order the different versions into a graph can result in poor provenance reconstructions [11,52]. Therefore, it becomes pertinent to utilize other sources of data to determine connections.…”

Generate, Segment and Refine: Towards Generic Manipulation Segmentation

“…nipulated images. Scouring the internet for "real" tampered images [24] is a laborious process that often also leads to over-fitting in the training process. Alternatively, one could employ a self-supervised process, where detected objects in one image are spliced onto another, with the caveat that such a process often results in training images that are not realistic.…”