Deep Domain Adaptation Approach for Classification of Disaster Images

Khattar, Anuradha; Quadri, S. M. K.

doi:10.1007/978-981-15-9509-7_21

Cited by 6 publications

(2 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Since labeling is a tedious and expensive task, self-labeling [47], [48], synthetic labeling [49], and semi-supervised learning [50] methods have also been proposed recently. At the onset of a disaster, the unavailability of labeled data has also encouraged researchers to propose methods based on transfer learning and domain adaptation [16], [51]. Li et al were among the first few researchers to explore this area.…”

Section: B Unimodal: Based On Image-only Modalitymentioning

confidence: 99%

“…Although several artificial intelligence-based tools have been proposed recently to make sense of this enormous crisis data and filter out relevant messages, most of these methods are based on a single modality. For example, these methods have independently used text [9], [10], [11], [12], images [13], [14], [15], [16], or videos [17] posted on social media platforms but have not fully explored recent multimodal techniques to exploit the complementary information provided by more than one modality.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CAMM: Cross-Attention Multimodal Classification of Disaster-Related Tweets

Khattar

Quadri

2022

IEEE Access

Self Cite

View full text Add to dashboard Cite

During the past decade, social media platforms have been extensively used during a disaster for information dissemination by the affected community and humanitarian agencies. Although many studies have been done recently to classify the informative and non-informative messages from social media posts, most are unimodal, i.e., have independently used textual or visual data to build the deep learning models. In the present study, we integrate the complementary information provided by the text and image messages about the same event posted by the affected community on the social media platform Twitter and build a multimodal deep learning model based on the concept of attention mechanism. The attention mechanism is a recent breakthrough that has revolutionized the field of deep learning. Just as humans pay more attention to a specific part of the text or image, ignoring the rest, neural networks can also be trained to concentrate on more relevant features through the attention mechanism. We propose a novel Cross-Attention Multi-Modal (CAMM) deep neural network for classifying multimodal disaster data, which uses the attention mask of the textual modality to highlight the features of the visual modality. We compare CAMM with unimodal models and the most popular bilinear multimodal models, MUTAN and BLOCK, generally used for visual question answering. CAMM achieves an average F1-score of 84.08%, better than the MUTAN and BLOCK methods by 6.31% and 5.91%, respectively. The proposed crossattention-based multimodal deep learning method outperforms the current state-of-the-art fusion methods on the benchmark multimodal disaster dataset by highlighting the more relevant cross-domain features of text and image tweets. This study affirms that social media platforms become a rich source of multimodal data during a disaster. This data can be utilized to build automated tools for quick filtration of informative messages to assess the post-disaster needs of the affected community and provide timely help.

show abstract