Accuracy of a pre-trained sentiment analysis (SA) classification model on tweets related to emergency response and early recovery assessment: the case of 2019 Albanian earthquake

Abstract

Traditionally, earthquake impact assessments have been made via fieldwork by non-governmental organisations (NGO's) sponsored data collection; however, this approach is time-consuming, expensive and often limited. Recently, social media (SM) has become a valuable tool for quickly collecting large amounts of first-hand data after a disaster and shows great potential for decision-making. Nevertheless, extracting meaningful information from SM is an ongoing area of research. This paper tests the accuracy of the pre-trained sentiment analysis (SA) model developed by the no-code machine learning platform MonkeyLearn using the text data related to the emergency response and early recovery phase of the three major earthquakes that struck Albania on the 26th November 2019. These events caused 51 deaths, 3000 injuries and extensive damage. We obtained 695 tweets with the hashtags: #Albania #AlbanianEarthquake, and #albanianearthquake from the 26th November 2019 to the 3rd February 2020. We used these data to test the accuracy of the pre-trained SA classification model developed by MonkeyLearn to identify polarity in text data. This test explores the feasibility to automate the classification process to extract meaningful information from text data from SM in real-time in the future. We tested the no-code machine learning platform's performance using a confusion matrix. We obtained an overall accuracy (ACC) of 63% and a misclassification rate of 37%. We conclude that the ACC of the unsupervised classification is sufficient for a preliminary assessment, but further research is needed to determine if the accuracy is improved by customising the training model of the machine learning platform.

Tweet Trends and Emotional Reactions: A Comprehensive Sentiment Analysis of Twitter Responses to the Earthquake in Turkey and Syria

“…Another study using the data from women living in Kocaeli (Turkey), investigated women's earthquake risk perception and factors helping to predict it by employing ordinary least square methods and revealed that the items that measure fear and financial perception have the highest mean among women's affective and cognitive risk perception factors [25]. The article [26] tested the accuracy of the pre-trained sentiment analysis (SA) model developed by the no-code machine learning platform MonkeyLearn through a confusion matrix where an overall accuracy (ACC) rate and a misclassification rate were 63% and 37% respectively using the text data of the three major earthquakes of Albania in 26th November 2019. Sentiment analysis has also gained attention during the analysis of the Covid-19 pandemic situation.…”