Visual and Textual Analysis for Image Trustworthiness Assessment within Online News

Abstract

The majority of news published online presents one or more images or videos, which make the news more easily consumed and therefore more attractive to huge audiences. As a consequence, news with catchy multimedia content can be spread and get viral extremely quickly. Unfortunately, the availability and sophistication of photo editing software are erasing the line between pristine and manipulated content. Given that images have the power of bias and influence the opinion and behavior of readers, the need of automatic techniques to assess the authenticity of images is straightforward. This paper aims at detecting images published within online news that have either been maliciously modified or that do not represent accurately the event the news is mentioning. The proposed approach composes image forensic algorithms for detecting image tampering, and textual analysis as a verifier of images that are misaligned to textual content. Furthermore, textual analysis can be considered as a complementary source of information supporting image forensics techniques when they falsely detect or falsely ignore image tampering due to heavy image postprocessing. The devised method is tested on three datasets. The performance on the first two shows interesting results, with F1-score generally higher than 75%. The third dataset has an exploratory intent; in fact, although showing that the methodology is not ready for completely unsupervised scenarios, it is possible to investigate possible problems and controversial cases that might arise in real-world scenarios.

Media forensics on social media platforms: a survey

“…Visual features from multimedia forensics are also extracted (namely double JPEG features [58], grid artifacts features [67], and Error Level Analysis 6 ) and jointly processed through logistic regressors and random forest classifiers. This approach has been extended in [68] by incorporating textual features used in sentiment analysis, and in [69] by exploiting additional advanced forensic visual features provided by the Splicebuster tool [70]. Moreover, these works are tested on datasets containing different kinds of composite objects, such as Tweets, news articles collected on Buzzfeed and Google News.…”

ARCNN framework for multimodal infodemic detection

“… Jin et al (2017) proposed using an attention mechanism for fusing visual features and concatenation for final classification. Lago et al (2019) applied a weighted late fusion approach that assigns weights w1 and w2 to text and image classification probabilities by calculating their product and later adding them. The details and mathematical background of these fusion mechanisms are explained in detail in Section 4.3 .…”

Multimodal Detection of Information Disorder from Social Media

“…Table I lists different approaches developed for information disorder detection, including approaches for mis-and disinformation detection, rumor verification, and fake news detection. The related literature can be split into two groups, monomodal approaches [3]- [5] and multimodal approaches [2], [6]- [12].…”

“…Mohtarami et al [4] propose a text-based method based on similarity modeling and stance filtering, which can extract text portions that can explain the factuality of a given claim. Lago et al [5] propose and evaluated different methods for identifying manipulated images in a dataset by using image forensic methods, moving the focus from textual to visual data. [4] x Lago et al (2019) [5] x Ruchansky et al (2017) [10] x x Zubiaga et al (2017) [9] x x Dong et al (2018) [8] x x Wang et al (2018) [7] x x Singhal et al (2019) [6] x x Nakamura et al (2020) [2] x x Jin et al (2017) [12] x x x Cui et al (2019) [11] x x x Papadopoulou et al (2019) [13] x x x…”