Automatic analysis of social media images to identify disaster type and infer appropriate emergency response

Abstract

Social media postings are increasingly being used in modern days disaster management. Along with the textual information, the contexts and cues inherent in the images posted on social media play an important role in identifying appropriate emergency responses to a particular disaster. In this paper, we proposed a disaster taxonomy of emergency response and used the same taxonomy with an emergency response pipeline together with deep-learning-based image classification and object identification algorithms to automate the emergency response decision-making process. We used the card sorting method to validate the completeness and correctness of the disaster taxonomy. We also used VGG-16 and You Only Look Once (YOLO) algorithms to analyze disaster-related images and identify disaster types and relevant cues (such as objects that appeared in those images). Furthermore, using decision tables and applied analytic hierarchy processes (AHP), we aligned the intermediate outputs to map a disaster-related image into the disaster taxonomy and determine an appropriate type of emergency response for a given disaster. The proposed approach has been validated using Earthquake, Hurricane, and Typhoon as use cases. The results show that 96% of images were categorized correctly on disaster taxonomy using YOLOv4. The accuracy can be further improved using an incremental training approach. Due to the use of cloud-based deep learning algorithms in image analysis, our approach can potentially be useful to real-time crisis management. The algorithms along with the proposed emergency response pipeline can be further enhanced with other spatiotemporal features extracted from multimedia information posted on social media.

Object Classification and Tracking Using Scaled P8 YOLOv4 Lite Model

“…Srivastava et al [16] compared the most recent and sophisticated CNN-based object detection techniques and concluded thatYOLOv3 shows the best overall performance. In order to categorize different sorts of crises and deduce the necessary emergency measures, Asif et al [17] offered an automatic analysis of social media images. The YOLOv2 Convolutional Neural Network was used by Saponara et al [19] to demonstrate real-time video-based fire and smoke detection in antifire surveillance systems.…”

A Citizen Science Approach for Analyzing Social Media With Crowdsourcing

“…One of the main goals of the research in this area is to automatically extract information to characterize, highlight and react to emergencies. This domain particularly seeks visual information because it provides substantial evidence of ongoing events [5], [24].…”

Analyzing social media with crowdsourcing in Crowd4SDG

“…One of the main goals of the research in this area is to automatically extract information about an event to characterize, highlight and react to emergencies. This domain particularly seeks visual information because it provides substantial evidence of ongoing events [5,21].…”

“…Therefore, filtering mechanisms must be put in place to prepare the dataset, clean the data (e.g., excluding exact duplicates and near duplicates), and filtering only potentially relevant data. Several models for image and text classification in social media have been developed, mainly based on deep neural networks (e.g., [5]). One of the open problems, beyond the development of the classifiers themselves, is the selection and configuration of the classifiers that are better suited for a given situation, as events can present distinctive characteristics in different contexts and locations around the world.…”

“…In particular, in Crowd4SDG, we focused on SDG 13 Climate action and on indicators to reduce the impact of emergency events on the population. The project studied different modalities to involve citizens and developed an approach to collect and refine ideas, proposing challenges on specific themes for citizens 5 . A set of challenges was proposed each year of the project, focusing on the combination of SDG 13 with a second SDG (SDG 11, SDG 5, and SDG 16 respectively), inviting citizen scientists to propose project ideas and helping refine them through an idea development program in which both project proposals were refined and tools for their realization were evaluated.…”