Development of a COVID-19–Related Anti-Asian Tweet Data Set: Quantitative Study (Preprint)

Abstract

BACKGROUND Since the advent of the COVID-19 pandemic, individuals of Asian descent have been the subject of stigma and hate speech in both offline and online communities. One of the major venues for encountering such unfair attacks is social networks such as Twitter. As the research community seeks to understand, analyze and implement detection techniques, high-quality datasets are becoming immensely important. OBJECTIVE In this study, we introduce a manually labeled dataset of Tweets having anti-Asian stigmatizing content. METHODS We sampled over 668M Tweets posted on Twitter between January 2020 to July 2020 and used an iterative data construction approach that includes three different stages of algorithm-driven data selection and manual labeling to finally arrive at 11,263 Tweets with primary labels (unknown/irrelevant, not-stigmatizing, stigmatizing-low, stigmatizing-medium, stigmatizing-high) and Tweet sub-topics (for e.g., wet market and eating habits, COVID-19 cases, bioweapon, etc.). Moreover, we selected 5,000 Tweets from that dataset and labeled them by a second annotator, and then a third annotator resolved conflicts in labels between first and second annotators. We present this final dataset as a high quality Twitter dataset on stigma towards Chinese people during COVID-19 pandemic. The dataset and instructions for labeling can be viewed in the Github repository: https://anonymous.4open.science/r/COVID-Stigma-A-Dataset-of-Anti-Asian-Stigmatizing-Tweets-During-COVID-19-65DD. RESULTS We implement some state-of-the-art models to detect stigmatizing Tweets to set initial benchmarks for our dataset. Our results show the Bidirectional Encoder Representations from Transformers (BERT) model achieves the highest accuracy of 79% when detecting stigma on unseen data with traditional models such as Support Vector Machine performing at 73% accuracy. CONCLUSIONS Our dataset can be used as a benchmark for further qualitative and quantitative research and analysis around the issue. We believe this contribution will help to significantly predict and hence reduce the unfair stigma, hate, and discrimination against Asian people during future crises like COVID-19.