Deep Multimodal Image-Repurposing Detection

Sabir, Ekraam; AbdAlmageed, Wael; Wu, Yue; Natarajan, Prem

doi:10.1145/3240508.3240707

Cited by 43 publications

(79 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The reuse of unmanipulated images to spread misinformation about a possibly unrelated entity or event was introduced in [8] as the verification of semantic integrity of data with image assets. Image repurposing with manipulated textual and location data has been studied more specifically in [24]. Our work falls in this category and we propose a novel generalized framework for image repurposing detection.…”

Section: Related Workmentioning

confidence: 99%

“…Previous works in this domain [8,24] focused on the detection of metadata modalities that are continuous in nature, such as captions and GPS coordinates in the form of latent encodings. The method of [8] is not suitable for structured metadata because it relies on learning a joint representation of images and captions.…”

Section: Baseline and State-of-the-art Modelsmentioning

confidence: 99%

“…The method of [8] is not suitable for structured metadata because it relies on learning a joint representation of images and captions. The publicly available deep multitask model (DMM) of [24] 10 was evaluated and we report the scores of this model on the IJBC-IRD and Painter by Numbers datasets. The package-similarity based retrieval used in their framework was not feasible on the Google Landmarks datasets, which has 1.2 million images.…”

Section: Baseline and State-of-the-art Modelsmentioning

confidence: 99%

“…However, the sheer volume and rate at which information is being created and disseminated necessitate developing automated ways of validating information. Several methods have, hence, been developed recently for detecting rumors on online forums [5,11,14,16,27,23], digital manipulations of images [3,28,29,30], and semantic inconsistencies in multimedia data [8,24]. While the detection of digital manipulation has gained most of the research attention over the years, rumor detection and semantic integrity verification are much newer areas of research.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

AIRD: Adversarial Learning Framework for Image Repurposing Detection

Jaiswal¹,

Wu²,

AbdAlmageed³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

Image repurposing is a commonly used method for spreading misinformation on social media and online forums, which involves publishing untampered images with modified metadata to create rumors and further propaganda. While manual verification is possible, given vast amounts of verified knowledge available on the internet, the increasing prevalence and ease of this form of semantic manipulation call for the development of robust automatic ways of assessing the semantic integrity of multimedia data. In this paper, we present a novel method for image repurposing detection that is based on the real-world adversarial interplay between a bad actor who repurposes images with counterfeit metadata and a watchdog who verifies the semantic consistency between images and their accompanying metadata, where both players have access to a reference dataset of verified content, which they can use to achieve their goals. The proposed method exhibits state-of-the-art performance on location-identity, subject-identity and painting-artist verification, showing its efficacy across a diverse set of scenarios.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Baseline and State-of-the-art Modelsmentioning

confidence: 99%

Section: Baseline and State-of-the-art Modelsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

AIRD: Adversarial Learning Framework for Image Repurposing Detection

Jaiswal¹,

Wu²,

AbdAlmageed³

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The first considers pairs of images and captions [4], and then trains a network to decide how consistent the caption is with the associated image. The second work [5] altered GPS coordinates, captions, and the actual image pixels for image re-purposing detection. In contrast to these methods, our approach operates only at the image pixel level and does not need any metadata, which may not be always available at hand.…”

Section: Related Workmentioning

confidence: 99%

Deep Learning Methods for Event Verification and Image Repurposing Detection

Goebel¹,

Flenner²,

Nataraj³

et al. 2019

View full text Add to dashboard Cite

The authenticity of images posted on social media is an issue of growing concern. Many algorithms have been developed to detect manipulated images, but few have investigated the ability of deep neural network based approaches to verify the authenticity of image labels, such as event names. In this paper, we propose several novel methods to predict if an image was captured at one of several noteworthy events. We use a set of images from several recorded events such as storms, marathons, protests, and other large public gatherings. Two strategies of applying pre-trained Imagenet network for event verification are presented, with two modifications for each strategy. The first method uses the features from the last convolutional layer of a pre-trained network as input to a classifier. We also consider the effects of tuning the convolutional weights of the pre-trained network to improve classification. The second method combines many features extracted from smaller scales and uses the output of a pre-trained network as the input to a second classifier. For both methods, we investigated several different classifiers and tested many different pre-trained networks. Our experiments demonstrate both these approaches are effective for event verification and image re-purposing detection. The classification at the global scale tends to marginally outperform our tested local methods and fine tuning the network further improves the results.

show abstract

A novel forgery classification method based on multi‐scale feature capsule network in mobile edge computing

Lian

Wang

2023

Softw Pract Exp

View full text Add to dashboard Cite

Face recognition is one of the most important applications of MEC. However, there have been many fake face data that can deceive MEC devices, causing serious problems such as information leakage. Face forgery detection can effectively solve this problem. Current face forgery detection methods have achieved high accuracy. However, most of the methods are researched on the classification of face authenticity. We find that studying multi‐classification of forgery methods can not only improve the accuracy of the model to identify fake faces, but also help improve the generalization ability of fake face classification. We argue that multi‐scale features and high‐frequency features can expose more detailed forgery artifacts. So, we design four modules, which take advantage of the complementarity of RGB features and frequency features, global features and local features. The first module is a residual‐guided multi‐scale spatial attention module, which uses residuals to guide the RGB feature extractor to extract fake features from a multi‐scale perspective. The second module is a multi‐scale retinal feature extraction module. The third module is a multi‐scale channel attention‐guided local frequency statistics module, which extracts local frequency responses using sliding‐window DCT. The last module is a capsule network classification module with overall correlation to classify the fused features. The information transfer between the subject capsule and the classification capsule can increase the integrity of the model, making the model converge faster. We conduct experiments on the databases FaceForensics++, DeepfakeDetection, and FakeAVCeleb. Experimental result shows that our method performs well on forgery classification.

show abstract

Deep Multimodal Image-Repurposing Detection

Cited by 43 publications

References 18 publications

AIRD: Adversarial Learning Framework for Image Repurposing Detection

AIRD: Adversarial Learning Framework for Image Repurposing Detection

Deep Learning Methods for Event Verification and Image Repurposing Detection

A novel forgery classification method based on multi‐scale feature capsule network in mobile edge computing

Contact Info

Product

Resources

About