Sentiment Analysis of Users’ Reviews on COVID-19 Contact Tracing Apps with a Benchmark Dataset (Preprint)

Abstract

BACKGROUND Contact tracing has been globally adopted in the fight to control the infection rate of COVID-19. Thanks to digital technologies, such as smartphones and wearable devices, contacts of COVID-19 patients can be easily traced and informed about their potential exposure to the virus. To this aim, several interesting mobile applications have been developed. However, there are ever-growing concerns over the working mechanism and performance of these applications. The literature already provides some interesting exploratory studies on the community’s response to the applications by analyzing information from different sources, such as news and users’ reviews of the applications. However, to the best of our knowledge, there is no existing solution that automatically analyzes users’ reviews and extracts the evoked sentiments. OBJECTIVE In this paper, we analyze how AI models can help in automatically extract and classify the polarity of users’ sentiments and propose a sentiment analysis framework to automatically analyze users’ reviews on COVID-19 contact tracing mobile applications. METHODS we propose a pipeline starting from manual annotation via a crowd-sourcing study and concluding on the development and training of AI models for automatic sentiment analysis of users’ reviews. In detail, we collected and annotated a large-scale dataset of Android and iOS mobile application users’ reviews for COVID-19 contact tracing. After manually analyzing and annotating users’ reviews, we employed both classical (i.e., Naïve Bayes, SVM, Random Forest) and deep learning (i.e., fastText, and different transformers) methods for classification experiments. This resulted in eight different classification models. RESULTS We employed eight different methods on three different tasks achieving up to an average F1-Scores 94.8% indicating the feasibility of automatic sentiment analysis of users’ reviews on the COVID-19 contact tracing applications. Moreover, the crowd-sourcing activity resulted in a large-scale benchmark dataset composed of 34,534 reviews manually annotated from the contract tracing applications of 46 distinct countries. CONCLUSIONS The existing literature mostly relies on the manual/exploratory analysis of users’ reviews on the application, which is a tedious and time-consuming process. Moreover, in the existing studies, generally, data from fewer applications are analyzed. In this work, we showed that automatic sentiment analysis can help in analyzing users’ responses to the application more quickly with significant accuracy. Moreover, we also provided a large-scale benchmark dataset composed of 34,534 reviews from 47 different applications. We believe the presented analysis and the dataset will support future research on the topic.

Reasons for Nonuse, Discontinuation of Use, and Acceptance of Additional Functionalities of a COVID-19 Contact Tracing App: Cross-sectional Survey Study

“…Moreover, previous research analyzing media content concerning the implementation of CTAs has identified thresholds and challenges experienced by users and showed the need to intensify communication about the benefits of using the apps [ 48 ]. By scraping social media and analyzing app users’ reviews, comments, and reported technical issues, developers could collect input to address reported issues and further develop CTA functionalities [ 4 , 49 ]. Also, by conducting in-depth interviews with potential users and analyzing media coverage on CTAs, the framing of the app’s functionalities and discussed issues can be detected [ 50 ].…”

Global User-Level Perception of COVID-19 Contact Tracing Applications: Data-Driven Approach Using Natural Language Processing

“…We made the data set publicly available for researchers to further explore the potential of NLP techniques for the automatic analysis of user feedback on contact tracing applications [ 29 ].…”

Klasifikasi Sentimen Ulasan Pengguna Aplikasi PeduliLindungi di Google Play Menggunakan Algoritma Support Vector Machine dengan Seleksi Fitur Chi-Square

“…Contact tracing yang diterapkan pada awalnya bersifat manual tracing dimana ketika orang yang terinfeksi virus memberikan informasi dengan siapa dan dimana ia melakukan kontak fisik dengan orang lain. Namun, penerapan manual tracing ini membuang banyak waktu dan sumber daya [2]. Dalam perkembangan teknologi pada saat ini, teknologi seperti smartphone dapat dimanfaatkan untuk membantu pemerintah melakukan tracing.…”