Crisis-DIAS: Towards Multimodal Damage Analysis - Deployment, Challenges and Assessment

Abstract

In times of a disaster, the information available on social media can be useful for several humanitarian tasks as disseminating messages on social media is quick and easily accessible. Disaster damage assessment is inherently multi-modal, yet most existing work on damage identification has focused solely on building generic classification models that rely exclusively on text or image analysis of online social media sessions (e.g., posts). Despite their empirical success, these efforts ignore the multi-modal information manifested in social media data. Conventionally, when information from various modalities is presented together, it often exhibits complementary insights about the application domain and facilitates better learning performance. In this work, we present Crisis-DIAS, a multi-modal sequential damage identification, and severity detection system. We aim to support disaster management and aid in planning by analyzing and exploiting the impact of linguistic cues on a unimodal visual system. Through extensive qualitative, quantitative and theoretical analysis on a real-world multi-modal social media dataset, we show that the Crisis-DIAS framework is superior to the state-of-the-art damage assessment models in terms of bias, responsiveness, computational efficiency, and assessment performance.

Sentiment Analysis of Fashion Related Posts in Social Media

“…In the past few years, multimodal sentiment analysis has attracted much attention including several tasks such as hate speech detection [12], emotion recognition [16,44], social media crisis handling [1,2], etc. For the lack of large-scaled dataset, datasets such as MOUD [30] MOSI [43], and MOSEI [44] are constructed based on product review and recommendation videos from YouTube, each associated with a sentiment label.…”

Robust Training of Social Media Image Classification Models for Rapid Disaster Response

“…The exploration of multimodality has also received attention in the research community [2,1]. In [2], authors explore different fusion strategies for multimodal learning.…”

“…The exploration of multimodality has also received attention in the research community [2,1]. In [2], authors explore different fusion strategies for multimodal learning. Similarly, in [1] a cross-attention based network exploited for multimodal fusion.…”

“…Very recently the study by Alam et al [9] developed a benchmark dataset, 1 which is consolidated from existing publicly available resources. The development process of this dataset consists of data curation from different existing sources, development of new data for new tasks, creating non-overlapping 2 training, development, and test sets. The reported benchmark dataset targeted four tasks mentioned earlier.…”

Exploring Spatiotemporal Change of City and Village from Remote Sensing Using Multi-Branch Networks

“…However, to further enhance the perceptions ability of human for urban and villages form , environmental and infrastructure, Wang, Tianyi and Tao, Yudong et.al [33] present a new multi-task and multi-modal deep learning framework with automatic loss weighting to assess damage after disaster events. Agarwal, Mansi and Leekha et.al [34] proposed a towards multimodal damage analysis methods to reply deployment, challenges and assessment and are called Crisis-DIAS. In order to promote human response to disaster events and extract as much detailed information as possible from limited data, Alam, Firoj and Ofli, Ferda et al [35] collected a large amount of multi-modal data (include image and text) from Twitter, effectively solving the limitation of lack of labeled image data, and improving The ability to respond to and manage disasters.…”