Modeling Intra and Inter-modality Incongruity for Multi-Modal Sarcasm Detection

Abstract

Sarcasm is a pervasive phenomenon in today's social media platforms such as Twitter and Reddit. These platforms allow users to create multi-modal messages, including texts, images, and videos. Existing multi-modal sarcasm detection methods either simply concatenate the features from multi modalities or fuse the multi modalities information in a designed manner. However, they ignore the incongruity character in sarcastic utterance, which is often manifested between modalities or within modalities. Inspired by this, we propose a BERT architecture-based model, which concentrates on both intra and inter-modality incongruity for multi-modal sarcasm detection. To be specific, we are inspired by the idea of self-attention mechanism and design intermodality attention to capturing inter-modality incongruity. In addition, the co-attention mechanism is applied to model the contradiction within the text. The incongruity information is then used for prediction. The experimental results demonstrate that our model achieves state-of-the-art performance on a public multi-modal sarcasm detection dataset.

FiLMing Multimodal Sarcasm Detection with Attention

“…The multimodal work that closely matches with our work is [10]. In that, the author proposes a BERT-based architecture for modeling intra-and intermodality incongruity.…”

“…Specifically, our model gives an improvement of 6.14% on F1-score and 5.15% on accuracy over the current SOTA from Bridge-RoBERTa model, thus verifying the effectiveness of our model. 0.8060 0.7797 0.8342 0.8402 Res-Bert [10] 0.8157 0.7887 0.8446 0.8480 IIMI-MMSD [10] 0.8292 0.8087 0.8508 0.8605 Bridge-RoBERTa [19] 0.8605 0.8295 0.8939 0.8851 Our Method 0.9219 0.9056 0.9387 0.9366…”

“…Intra-modality incongruity is also used in [20] for sarcasm detection. Contrary to this work, we model inter-modality incongruity between the text and visual attributes (The work in [10] uses text and visual features) in two ways. The first uses visual attributes and text, and the latter uses visual features and text.…”

“…Then, they use a Decomposition and Relation Network (D&R Net) to model cross-modality contrast using ANP and semantic association between the image and text. -Res-bert: [10] implements Res-bert as a model to concatenate the output of image features from ResNet and text features from BERT. Since this model closely resembles our approach, it is an important baseline.…”

“…Since this model closely resembles our approach, it is an important baseline. -Intra and Inter-modality Incongruity (IIMI-MMSD): [10] proposes a BERT-based model, which concentrates on both intra and inter-modality incongruity for multimodal sarcasm detection. They use self-attention and co-attention mechanism to capture inter and intra-modality incongruity, respectively.…”