FMFN: Fine-Grained Multimodal Fusion Networks for Fake News Detection

Abstract

As one of the most popular social media platforms, microblogs are ideal places for news propagation. In microblogs, tweets with both text and images are more likely to attract attention than text-only tweets. This advantage is exploited by fake news producers to publish fake news, which has a devasting impact on individuals and society. Thus, multimodal fake news detection has attracted the attention of many researchers. For news with text and image, multimodal fake news detection utilizes both text and image information to determine the authenticity of news. Most of the existing methods for multimodal fake news detection obtain a joint representation by simply concatenating a vector representation of the text and a visual representation of the image, which ignores the dependencies between them. Although there are a small number of approaches that use the attention mechanism to fuse them, they are not fine-grained enough in feature fusion. The reason is that, for a given image, there are multiple visual features and certain correlations between these features. They do not use multiple feature vectors representing different visual features to fuse with textual features, and ignore the correlations, resulting in inadequate fusion of textual features and visual features. In this paper, we propose a novel fine-grained multimodal fusion network (FMFN) to fully fuse textual features and visual features for fake news detection. Scaled dot-product attention is utilized to fuse word embeddings of words in the text and multiple feature vectors representing different features of the image, which not only considers the correlations between different visual features but also better captures the dependencies between textual features and visual features. We conduct extensive experiments on a public Weibo dataset. Our approach achieves competitive results compared with other methods for fusing visual representation and text representation, which demonstrates that the joint representation learned by the FMFN (which fuses multiple visual features and multiple textual features) is better than the joint representation obtained by fusing a visual representation and a text representation in determining fake news.

Multimodal fake news detection using hyperparameter-tuned BERT and ResNet110

“…Wang et al [25] implemented a fine-grained multimodal fusion network (FMFN) as a nuanced method for the fusion of textual and visual features for effective FND. The suggested approach used scaled dot-product attention method to enhance both features and fuse the improved features, thereby capturing the province among features.…”

How Risky is Multimodal Fake News Detection? A Review of Cross-Modal Learning Approaches under EU AI Act Constrains

“…Year Method COOLANT [16] 2023 Cross-modal contrastive learning MMFN [4] 2023 Multi-grained information fusion MPFN [17] 2023 Multimodal progressive fusion SAMPLE [18] 2023 Similarity-aware multimodal prompt learning TieFake [19] 2023 Integration of multimodal context and author sentiment: focusing on title-text similarity and emotion awareness FNR [20] 2023 Similarity and transformer-based learning DGM [21] 2023 Transformer based on manipulation-aware contrastive learning and modality-aware cross-attention CAFE [22] 2022 Cross-modal ambiguity learning CMC [23] 2022 Cross-modal knowledge distillation FND-CLIP [24] 2022 Contrastive language-image pretraining-guided learning LIIMR [25] 2022 Leveraging intra and inter modality relationship FMFN [26] 2022 Fine-grained multimodal fusion network MCAN [27] 2021 Multimodal co-attention networks AMFB [28] 2021 Attention-based multimodal factorized bilinear pooling HMCAN [29] 2021 Hierarchical multi-modal contextual attention network CARMN [30] 2021 Crossmodal attention residual and multichannel CNN SAFE [31] 2020 Cross-modal similarity measurement stages or layers of a neural network. The main idea is to enable the model to progressively integrate information from different modalities, instead of attempting to do so in a single step.…”

“…AMFB [28] proposes an attention-based stacked bi-directional LSTM network that captures textual information at different levels, attentionbased multi-level convolutional neural network-recurrent neural network for visual feature extraction, multimodal factorized bilinear pooling to combine the textual and visual feature representations, and then passes them through a multi-layer perceptron for FND. Wang et al [26] introduce a Fine-grained Multimodal Fusion Network (FMFN) to fuse textual and visual features fully. The proposed model utilizes scaled dotproduct attention mechanisms for the fine-grained fusion of the textual and the visual features, which not only takes into account the correlations between different visual features but also captures the dependencies between textual and visual features.…”

Multi-modal Misinformation Detection: Approaches, Challenges and Opportunities

“…TRANSFAKE adopts a preprocessing method similar to BERT for concatenated text, comment and image. In another work (Wang, Mao, and Li 2022), Wang et al apply scaled dot-product attention on top of image and text features as a fine-grained fusion and use the fused feature to classify articles. Wang et al propose a deep learning network for Biomedical informatics that leverages visual and textual information and a semantic-and tasklevel attention mechanism to focus on the essential contents of a post that signal anti-vaccine messages (Wang, Yin, and Argyris 2021).…”