CAMM: Cross-Attention Multimodal Classification of Disaster-Related Tweets

Abstract

During the past decade, social media platforms have been extensively used during a disaster for information dissemination by the affected community and humanitarian agencies. Although many studies have been done recently to classify the informative and non-informative messages from social media posts, most are unimodal, i.e., have independently used textual or visual data to build the deep learning models. In the present study, we integrate the complementary information provided by the text and image messages about the same event posted by the affected community on the social media platform Twitter and build a multimodal deep learning model based on the concept of attention mechanism. The attention mechanism is a recent breakthrough that has revolutionized the field of deep learning. Just as humans pay more attention to a specific part of the text or image, ignoring the rest, neural networks can also be trained to concentrate on more relevant features through the attention mechanism. We propose a novel Cross-Attention Multi-Modal (CAMM) deep neural network for classifying multimodal disaster data, which uses the attention mask of the textual modality to highlight the features of the visual modality. We compare CAMM with unimodal models and the most popular bilinear multimodal models, MUTAN and BLOCK, generally used for visual question answering. CAMM achieves an average F1-score of 84.08%, better than the MUTAN and BLOCK methods by 6.31% and 5.91%, respectively. The proposed crossattention-based multimodal deep learning method outperforms the current state-of-the-art fusion methods on the benchmark multimodal disaster dataset by highlighting the more relevant cross-domain features of text and image tweets. This study affirms that social media platforms become a rich source of multimodal data during a disaster. This data can be utilized to build automated tools for quick filtration of informative messages to assess the post-disaster needs of the affected community and provide timely help.

DPHFM: A Deep Parallel Hybrid Fusion Model for Disaster Tweet Classification on Twitter Data

“…Deep learning, a subset of ML, has also found extensive applications in handling Twitter data due to its ability to automatically learn composite patterns and representations from large-scale data. In the latest literature mainly focused on deep networks, the authors [5] proposed a new deep neural network called Cross-Attention Multi-Modal (CAMM) to classify disaster data that contains both text and images. The authors [6] proposed a novel method, called Stacking-based Ensemble [43], using Statistical features and Informative words to tackle the challenges of damage assessment in tweets.…”

“…Continuous Bag of Words (CBOW): Based on the context words around a target word, CBOW attempts to anticipate that term. To maximize the chance of correctly predicting the target word "w_t" given its context words "w_c," where "c" spans from "-C" to "C" (excluding 0 as the target word itself), one must start with a context window of size "C" (number of words on each side of the target word) shown in equation [5] 𝑀𝑎𝑥𝑖𝑚𝑖𝑧𝑒 ( 1 𝑇 ) * Σ(Log P(w t |w c ))…”

“…In the case of Twitter, tweet information is collected using the Twitter API, API key, and Bear token [5], which can be obtained through Twitter APL. Relevant hashtags can then be used to collect tweet information.…”

“…In most of the literature on tweet classification, the datasets are initially created with manual annotations [8] so that the annotators can preprocess the individual tweets. Late the following recent trends, many researchers focused on machine learning based [3][4][5], deep learning based [11] approaches. In the recent past, most researchers have focused on transfer learning approaches such as voting [31,39], stacking [10,42], and BERT [33] to improve classification tasks.…”

Identification of Blue Horizontal Branch Stars with Multimodal Fusion

“…Multimodal learning is a general method for building artificial intelligence (AI) models that extract and correlate information from multimodal data (Baltrusaitis et al 2019). Multimodal learning has been used in several areas (Khattar & Quadri 2022), such as visual question answering, emotion recognition, machine translation, cross-modal retrieval, and speech recognition. With the development of large survey telescopes, a massive amount of multi-source heterogeneous astronomical data, such as spectral and photometric data of astronomical objects, have been generated.…”