AIDA-UPM at SemEval-2022 Task 5: Exploring Multimodal Late Information Fusion for Multimedia Automatic Misogyny Identification

Abstract

This paper describes the multimodal late fusion model proposed in the SemEval-2022 Multimedia Automatic Misogyny Identification (MAMI) task. The main contribution of this paper is the exploration of different late fusion methods to boost the performance of the combination based on the Transformer-based model and Convolutional Neural Networks (CNNs) for text and image, respectively. Additionally, our findings contribute to a better understanding of the effects of different image preprocessing methods for meme classification. We achieve 0.636 F1-macro average score for the binary sub-task A, and 0.632 F1-macro average score for the multi-label sub-task B. The present findings might help solve the inequality and discrimination women suffer on social media platforms.

DeVi Deep Learning Framework for Misogyny Identification in Multimodal Data

“…The proposed model uses the bilinear interaction layer to fuse the image and text features. Huertas et al [40] explored several late fusion techniques in order to enhance the effectiveness of the fusion approach utilising a model based on transformers and convolutional neural networks (CNNs). The proposed method encompasses a multimodal strategy that integrates several attributes (such as logits, probabilities, and embeddings) derived from both textual and visual elements, employing a late fusion technique.…”