I didn’t mean what I wrote! Exploring Multimodality for Sarcasm Detection

Akhtar, Md. Shad; Behera, Pranati; Ekbal, Asif

doi:10.1109/ijcnn48605.2020.9206905

Cited by 16 publications

(11 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since final predictions are made using features of the highlighted areas, this operation gives an overall boost to the model performance. Moreover, we can verify from Figure 3(a) and 3(b) that our model is able to attend to text like "pack of almonds" and "stupid people" without the need to explicitly use noisy Optical Character Recognition (OCR), an approach used in previous works [10,15].…”

Section: Model Analysismentioning

confidence: 77%

“…The work in [1] extracts visual features and visual attributes from images using ResNet and builds a hierarchical fusion model to detect sarcasm. Along the same lines, the recurrent network model in [15] proposes the idea of a gating mechanism to leak information from one modality to the other and achieves superior performance on Twitter benchmark dataset for sarcasm detection. The authors of [19] use pre-trained BERT and ResNet models to encode text and image data and connect the two using a gate called a bridge.…”

Section: Related Workmentioning

confidence: 99%

“…1(b), the textual description and image semantics are alluding to opposite meanings. Such a phenomena is called incongruity [4,20,13] and has been leveraged to tackle multimodal sarcasm detection [15,21,19,16]. Following previous multimodal sarcasm detection approaches, we propose a deep learning-based architecture that takes as input text and image modality.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

FiLMing Multimodal Sarcasm Detection with Attention

Gupta¹,

Shah²,

Shah³

et al. 2021

Preprint

View full text Add to dashboard Cite

Sarcasm detection identifies natural language expressions whose intended meaning is different from what is implied by its surface meaning. It finds applications in many NLP tasks such as opinion mining, sentiment analysis, etc. Today, social media has given rise to an abundant amount of multimodal data where users express their opinions through text and images. Our paper aims to leverage multimodal data to improve the performance of the existing systems for sarcasm detection. So far, various approaches have been proposed that uses text and image modality and a fusion of both. We propose a novel architecture that uses the RoBERTa model with a co-attention layer on top to incorporate context incongruity between input text and image attributes. Further, we integrate feature-wise affine transformation by conditioning the input image through FiLMed ResNet blocks with the textual features using the GRU network to capture the multimodal information. The output from both the models and the CLS token from RoBERTa is concatenated and used for the final prediction. Our results demonstrate that our proposed model outperforms the existing state-of-the-art method by 6.14% F1 score on the public Twitter multimodal sarcasm detection dataset.

show abstract

Section: Model Analysismentioning

confidence: 77%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

FiLMing Multimodal Sarcasm Detection with Attention

Gupta¹,

Shah²,

Shah³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…modal 12 textual and bi-modal textual+acoustic information. In text modality, SVM on T avg reports mediocre F1-scores of 22.5% and 35.6% for the sarcasm and humor classification, respectively.…”

Section: Comparative Analysismentioning

confidence: 99%

“…Thus, it offers an excellent opportunity to study the sarcasm or humor in a context. A few previous attempts [10,11,12] on sarcasm classification involved multi-modal information in a conversation to leverage the context and extract the incongruity between the surface and expressed semantics. Similarly, many studies [13,14] employed images and visual frames along with the text to detect humor.…”

Section: Introductionmentioning

confidence: 99%

Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations

Bedi,

Kumar,

Akhtar

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Sarcasm detection and humor classification are inherently subtle problems, primarily due to their dependence on the contextual and non-verbal information. Furthermore, existing studies in these two topics are usually constrained in non-English languages such as Hindi, due to the unavailability of qualitative annotated datasets. In this work, we make two major contributions considering the above limitations: (1) we develop a Hindi-English code-mixed dataset, MaSaC 1 , for the multi-modal sarcasm detection and humor classification in conversational dialog, which to our knowledge is the first dataset of its kind; (2) we propose MSH-COMICS 2 , a novel attention-rich neural architecture for the utterance classification. We learn efficient utterance representation utilizing a hierarchical attention mechanism that attends to a small portion of the input sentence at a time. Further, we incorporate dialog-level contextual attention mechanism to leverage the dialog history for the multi-modal classification. We perform extensive experiments for both the tasks by varying multi-modal inputs and various submodules of MSH-COMICS. We also conduct comparative analysis against existing approaches. We observe that MSH-COMICS attains superior performance over the existing models by >1 F1-score point for the sarcasm detection and 10 F1-score points in humor classification. We diagnose our model and perform thorough analysis of the results to understand the superiority and pitfalls.

show abstract

FiLMing Multimodal Sarcasm Detection with Attention

Gupta

Shah

et al. 2021

Communications in Computer and Information Science

View full text Add to dashboard Cite

I didn’t mean what I wrote! Exploring Multimodality for Sarcasm Detection

Cited by 16 publications

References 12 publications

FiLMing Multimodal Sarcasm Detection with Attention

FiLMing Multimodal Sarcasm Detection with Attention

Multi-modal Sarcasm Detection and Humor Classification in Code-mixed Conversations

FiLMing Multimodal Sarcasm Detection with Attention

Contact Info

Product

Resources

About