Multimedia Semantic Integrity Assessment Using Joint Embedding Of Images And Text

Abstract

Real world multimedia data is o en composed of multiple modalities such as an image or a video with associated text (e.g. captions, user comments, etc.) and metadata. Such multimodal data packages are prone to manipulations, where a subset of these modalities can be altered to misrepresent or repurpose data packages, with possible malicious intent. It is, therefore, important to develop methods to assess or verify the integrity of these multimedia packages. Using computer vision and natural language processing methods to directly compare the image (or video) and the associated caption to verify the integrity of a media package is only possible for a limited set of objects and scenes. In this paper, we present a novel deep learning-based approach for assessing the semantic integrity of multimedia packages containing images and captions, using a reference set of multimedia packages. We construct a joint embedding of images and captions with deep multimodal representation learning on the reference dataset in a framework that also provides image-caption consistency scores (ICCSs). e integrity of query media packages is assessed as the inlierness of the query ICCSs with respect to the reference dataset. We present the MultimodAl Information Manipulation dataset (MAIM), a new dataset of media packages from Flickr, which we make available to the research community. We use both the newly created dataset as well as Flickr30K and MS COCO datasets to quantitatively evaluate our proposed approach. e reference dataset does not contain unmanipulated versions of tampered query packages. Our method is able to achieve F 1 scores of 0.75, 0.89 and 0.94 on MAIM, Flickr30K and MS COCO, respectively, for detecting semantically incoherent media packages. ACM Reference format:

AIRD: Adversarial Learning Framework for Image Repurposing Detection

“…Figure 1 shows some real-world examples of image repurposing. Jaiswal et al [8] define the broader problem of multimedia semantic integrity assessment as the verification of consistency between the media asset (e.g. image) and different components of the associated metadata (e.g.…”

“…They also introduce the concept of using a reference dataset (RD) of untampered packages to assist the validation of query packages. Image repurposing detection falls under this umbrella and has been explored in packages of images and captions [8] as well as those that additionally contain Global Positioning System (GPS) information [24].…”

“…The reuse of unmanipulated images to spread misinformation about a possibly unrelated entity or event was introduced in [8] as the verification of semantic integrity of data with image assets. Image repurposing with manipulated textual and location data has been studied more specifically in [24].…”

“…Jaiswal et al [8] introduced the general approach of using a reference dataset of packages as a knowledge-base from which evidence could be gathered, to assist the semantic integrity assessment of a query package. Conceptually, this approach is similar to how watchdogs verify news articles online, with news sources, document repositories and encyclopedias on the internet serving as enormous reference datasets.…”

“…However, the sheer volume and rate at which information is being created and disseminated necessitate developing automated ways of validating information. Several methods have, hence, been developed recently for detecting rumors on online forums [5,11,14,16,27,23], digital manipulations of images [3,28,29,30], and semantic inconsistencies in multimedia data [8,24]. While the detection of digital manipulation has gained most of the research attention over the years, rumor detection and semantic integrity verification are much newer areas of research.…”