Deep Image Comparator: Learning to Visualize Editorial Change

Abstract

We present a novel architecture for comparing a pair of images to identify image regions that have been subjected to editorial manipulation. We first describe a robust near-duplicate search, for matching a potentially manipulated image circulating online to an image within a trusted database of originals. We then describe a novel architecture for comparing that image pair, to localize regions that have been manipulated to differ from the retrieved original. The localization ignores discrepancies due to benign image transformations that commonly occur during online redistribution. These include artifacts due to noise and recompression degradation, as well as out-of-place transformations due to image padding, warping, and changes in size and shape. Robustness towards out-of-place transformations is achieved via the end-to-end training of a differentiable warping module within the comparator architecture. We demonstrate effective retrieval and comparison of benign transformed and manipulated images, over a dataset of millions of photographs.

ARIA: Adversarially Robust Image Attribution for Content Provenance

“…1. We show that this algorithm improves adversarial robustness of both tamper-sensitive [46] and tamper-invariant [6] image fingerprinting models, and we also discuss how to make the image comparator model [6] robust. The approach is conceptually simple and leads to a relatively small computational overhead (≈ 2× slowdown) compared to standard contrastive learning.…”

“…Image fingerprinting for provenance. Image fingerprinting models robust to non-editorial transformations were proposed in Black et al [6] and Nguyen et al [46]. These represent two complementary approaches to applying image retrieval to the attribution problem.…”

“…However, the approaches differ in their consideration of manipulated images. Black et al [6] train the image retrieval model to match manipulated images to originals successfully, i.e., to bring such images pairs close together in the search embedding. By contrast, Nguyen et al [46] train the model to separate such image pairs, encouraging matching to fail in the presence of content manipulation.…”

“…Detection methods automatically identify manipulated or synthetic images through visual artifacts or statistics [60,61,66]. Attribution methods match an image to a trusted database of originals [5,6,46]. * Work done as internship at Adobe Research.…”

“…In such 'tamper-sensitive' matching, a manipulated image would not be falsely corroborated by provenance data associated with the original. By contrast, Black et al [6] learn a 'tamper-invariant' image fingerprint which is insensitive to both non-editorial and editorial change, and visually highlight manipulated changes using a separate model. This paper reports on the novel problem of adversarial attack and defense for these image attribution approaches.…”