Multi-Level Multi-Modal Cross-Attention Network for Fake News Detection

Abstract

With the development of the Mobile Internet, more and more users publish multi-modal posts on social media platforms. Fake news detection has become an increasingly challenging task. Although there are many works using deep schemes to extract and combine textual and visual representation in the post, most existing methods do not sufficiently utilize the complementary multi-modal information containing semantic concepts and entities to complement and enhance each modality. Moreover, these methods do not model and incorporate the rich multi-level semantics of text information to improve fake news detection tasks. In this paper, we propose a novel end-to-end Multi-level Multi-modal Cross-attention Network (MMCN) which exploits the multi-level semantics of text and jointly integrates the relationships of duplicate and different modalities (textual and visual modality) of social multimedia posts in a unified framework. Firstly, pre-trained BERT and ResNet models are employed to generate high-quality representations for text words and image regions, respectively. A multi-modal cross-attention network is then designed to fuse the feature embeddings of the text words and image regions by simultaneously considering data relationships in duplicate and different modalities. Specially, due to different layers of the transformer architecture have different feature representations, we employ a multi-level encoding network to capture the rich multi-level semantics to enhance the presentations of posts. Extensive experiments on the two public datasets (WEIBO and PHEME) demonstrate that compared with the state-of-the-art models, the proposed MMCN has an advantageous performance.

Combating multimodal fake news on social media: methods, datasets, and future perspective

“…Word2Vec is one of the popular word embedding that is used by [ 4 , 25 , 27 , 91 ]. But as word2vec can’t handle out-of-vocabulary words, researchers have exploited Glove, BERT, XLNet and other embeddings instead [ 26 , 28 , 92 , 95 , 96 , 98 , 110 , 129 ]. FND-SCTI [ 4 ] considers the hierarchical document structure and uses Bi-LSTM at both word-level (from word to sentence) and sentence-level (from sentence to document) to capture the long-term dependencies in the text.…”

“…FND-SCTI [ 4 ] considers the hierarchical document structure and uses Bi-LSTM at both word-level (from word to sentence) and sentence-level (from sentence to document) to capture the long-term dependencies in the text. Ying et al [ 92 ] proposed a multi-level encoding network to capture the multi-level semantics in the text. The model KMAGCN [ 99 ] proposed by Qian et al captures the non-consecutive and long-range semantic relations of the post by modeling it as a graph rather than a word sequence and proposes a novel adaptive graph convolutional network handle the variability in the graph data.…”

“…To filter out the noise and highlight the image regions that are strongly related to the target word, [ 16 ] uses a word-guided visual attention module. To learn complementary inter-dependencies among textual and visual features [ 26 , 92 , 96 , 99 , 129 ] uses multiple co-attention layers, hierarchical multi-modal contextual attention network, feature-level attention mechanism, blended attention module and multi-modal cross-attention network respectively. The proposed the Crossmodal Attention Residual Network (CARN) in [ 111 ] can selectively extract information pertaining to a target modality from another source modality while preserving the target modality's unique information.…”

“…Detecting fake news at the event level comprises of predicting the veracity of the whole event instead of an individual post. Most of the existing MFND frameworks work at post level [ 4 , 27 , 28 , 92 , 95 ], and learn event-specific features that are not useful in predicting the unseen future events and suffer from generalization. But [ 15 , 16 ] intends to accurately categorize the post into one of the K events based on the multimodal feature representations.…”

Beyond the Transformer: A Novel Polynomial Inherent Attention (PIA) Model and Its Great Impact on Neural Machine Translation

“…Transformers are currently state-of-the-art models in neural machine translation [1,2]. Since their introduction in 2017 [3], transformers have consistently produced state-of-the-art results in many areas of NLP and NMT [4][5][6][7][8][9][10][11][12][13][14]. However, based on recent results, it appears that transformers are reaching their limits.…”