Exercise? I thought you said 'Extra Fries’: Leveraging Sentence Demarcations and Multi-hop Attention for Meme Affect Analysis

Abstract

Today's Internet is awash in memes as they are humorous, satirical, or ironic which make people laugh. According to a survey, 33% of social media users in age bracket [13-35] send memes every day, whereas more than 50% send every week. Some of these memes spread rapidly within a very short time-frame, and their virality depends on the novelty of their (textual and visual) content. A few of them convey positive messages, such as funny or motivational quotes; while others are meant to mock/hurt someone's feelings through sarcastic or offensive messages. Despite the appealing nature of memes and their rapid emergence on social media, effective analysis of memes has not been adequately attempted to the extent it deserves. Recently, in SemEval'20, a pioneering attempt has been made in this direction by organizing a shared task on `Memotion Analysis' (meme emotion analysis). As expected, the competition attracted more than 500 participants with the final submission of [23-32] systems across three sub-tasks. In this paper, we attempt to solve the same set of tasks suggested in the SemEval'20 - Memotion Analysis competition. We propose a multi-hop attention-based deep neural network framework, called MHA-Meme, whose prime objective is to leverage the spatial-domain correspondence between the visual modality (an image) and various textual segments to extract fine-grained feature representations for classification. We evaluate MHA-Meme on the `Memotion Analysis' dataset for all three sub-tasks - sentiment classification, affect classification, and affect class quantification. Our comparative study shows state-of-the-art performances of MHA-Meme for all three tasks compared to the top systems that participated in the competition. Unlike all the baselines which perform inconsistently across all three tasks, MHA-Meme outperforms baselines in all the tasks on average. Moreover, we validate the generalization of MHA-Meme on another set of manually annotated test samples and observe it to be consistent. Finally, we establish the interpretability of MHA-Meme.

Meme Sentiment Analysis Enhanced with Multimodal Spatial Encoding and Face Embedding

“…Their findings suggest that meme classification architectures exhibit adaptability across different affective computing tasks. Furthermore, Pramanick et al [9], who reported the best-performing sentiment classification solution to the Memotion 1.0 dataset [13], showed that the same architecture outperforms all, or all but one, competing solution when individually trained on eight affect dimensions.…”

“…A typical approach to building a multimodal meme classifier is to generate unimodal representations of each modality before fusing these representations into a multimodal representation of the meme, such as in [3,10,12,9]. Furthermore, the literature presents a wide range of deep learning representations used for each visual and textual modality [6,7,13], with no clear evidence that any of the options would consistently outperform all others.…”

“…In this task, Pramanick et al [9] showed performance gains by segmenting the text modality into text clusters but did not explicitly represent the spatial position of each cluster. To classify hateful memes, Zhu [16] employed a patch detector to divide each meme into "image regions".…”

“…They attributed their model's outperforming of other leading hateful meme classifiers to its "awareness" of offensive intermodal analogies: the purposeful superimposing of a text cluster near to a visual object is used to represent an offensive conceptual comparison. While their approach is predicated solely on offensive spatial analogies, we posit that this approach could capture a broader category of intermodal spatial relationships, including those captured by Pramanick et al's [9] and Zhu's [16] approaches.…”

“…Finally, the Filtered-OCR datasets replace the text of each meme in Filtered with that returned in our feature extraction OCR step. Unlike in [7,9,13], we excluded any additional human validation during the OCR extraction process. All models are trained, validated, and tested on the resultant Filtered-OCR datasets.…”