Multimodal Analysis of Disaster Tweets

Gautam, Akash; Misra, Luv; Kumar, Ajit; Misra, Kush; Aggarwal, Shashwat; Shah, Rajiv Ratn

doi:10.1109/bigmm.2019.00-38

Cited by 37 publications

(15 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As shown in Table 9, the method using multimodality had better results than the method using unimodality. Compared to the model designed by Gautam [37] and Ofli [23], the architecture of the model designed in this paper was simpler and easier to train. Specifically, in the image feature extractor module, we adjusted the parameter size in the second to last layer of its fully connected layer to 500 from the original size of 1000, which means that the image feature is simpler and that the multimodal fusion input is simpler as well.…”

Section: Discussionmentioning

confidence: 99%

“…For example, Dao et al [36] showed a context-aware data-fusion method for disaster image retrieval from social media where the system combined the image with text. In 2019, using CNN, VGG, and long short-term memory(LSTM), Gautam et al [37] designed models with multimodal data to categorize the information found on Twitter, which could further improve the accuracy of the classification task. In 2020, Ofli et al [23] exploited both text and image modalities from social media and mined the useful disaster information from them.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Disaster Image Classification by Fusing Multimodal Social Media Data

Gan

Huang

Cai

et al. 2021

IJGI

View full text Add to dashboard Cite

Social media datasets have been widely used in disaster assessment and management. When a disaster occurs, many users post messages in a variety of formats, e.g., image and text, on social media platforms. Useful information could be mined from these multimodal data to enable situational awareness and to support decision making during disasters. However, the multimodal data collected from social media contain a lot of irrelevant and misleading content that needs to be filtered out. Existing work has mostly used unimodal methods to classify disaster messages. In other words, these methods treated the image and textual features separately. While a few methods adopted multimodality to deal with the data, their accuracy cannot be guaranteed. This research seamlessly integrates image and text information by developing a multimodal fusion approach to identify useful disaster images collected from social media platforms. In particular, a deep learning method is used to extract the visual features from social media, and a FastText framework is then used to extract the textual features. Next, a novel data fusion model is developed to combine both visual and textual features to classify relevant disaster images. Experiments on a real-world disaster dataset, CrisisMMD, are performed, and the validation results demonstrate that the method consistently and significantly outperforms the previously published state-of-the-art work by over 3%, with a performance improvement from 84.4% to 87.6%.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Disaster Image Classification by Fusing Multimodal Social Media Data

Gan

Huang

Cai

et al. 2021

IJGI

View full text Add to dashboard Cite

show abstract

“…Under the multimodal analysis, Gautam et al [25] proposed a diffusion method for the classification of Twitter data (text and images) of seven disasters of the CrisisMMD dataset [33] into two classes informative and noninformative and compared their model with the unimodal models based on text-only and image-only modalities. For text-only modality, they applied N-gram, LSTM, BiLSTM, and CNN+Glove methods, and for image-only modality, they used six pre-trained models VGG-16, VGG-19, ResNet50, InceptionV2, Xception, and DenseNet for transfer learning.…”

Section: Multimodal: Based On Both Text and Image Modalitiesmentioning

confidence: 99%

“…Under the multimodal category, the most common methods used by researchers to fuse the features of text and image modalities for the classification of disaster-related data are early fusion and late fusion [25], [26], [27], [28], [29]. Early fusion is also called feature-based fusion, where the final decision is based on the common vector obtained after concatenating the extracted features of the individual modalities.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

CAMM: Cross-Attention Multimodal Classification of Disaster-Related Tweets

Khattar

Quadri

2022

IEEE Access

View full text Add to dashboard Cite

During the past decade, social media platforms have been extensively used during a disaster for information dissemination by the affected community and humanitarian agencies. Although many studies have been done recently to classify the informative and non-informative messages from social media posts, most are unimodal, i.e., have independently used textual or visual data to build the deep learning models. In the present study, we integrate the complementary information provided by the text and image messages about the same event posted by the affected community on the social media platform Twitter and build a multimodal deep learning model based on the concept of attention mechanism. The attention mechanism is a recent breakthrough that has revolutionized the field of deep learning. Just as humans pay more attention to a specific part of the text or image, ignoring the rest, neural networks can also be trained to concentrate on more relevant features through the attention mechanism. We propose a novel Cross-Attention Multi-Modal (CAMM) deep neural network for classifying multimodal disaster data, which uses the attention mask of the textual modality to highlight the features of the visual modality. We compare CAMM with unimodal models and the most popular bilinear multimodal models, MUTAN and BLOCK, generally used for visual question answering. CAMM achieves an average F1-score of 84.08%, better than the MUTAN and BLOCK methods by 6.31% and 5.91%, respectively. The proposed crossattention-based multimodal deep learning method outperforms the current state-of-the-art fusion methods on the benchmark multimodal disaster dataset by highlighting the more relevant cross-domain features of text and image tweets. This study affirms that social media platforms become a rich source of multimodal data during a disaster. This data can be utilized to build automated tools for quick filtration of informative messages to assess the post-disaster needs of the affected community and provide timely help.

show abstract