Investigating User Radicalization: A Novel Dataset for Identifying Fine-Grained Temporal Shifts in Opinion

Abstract

There is an increasing need for the ability to model fine-grained opinion shifts of social media users, as concerns about the potential polarizing social effects increase. However, the lack of publicly available datasets that are suitable for the task presents a major challenge. In this paper, we introduce an innovative annotated dataset for modeling subtle opinion fluctuations and detecting fine-grained stances. The dataset includes a sufficient amount of stance polarity and intensity labels per user over time and within entire conversational threads, thus making subtle opinion fluctuations detectable both in long term and in short term. All posts are annotated by non-experts and a significant portion of the data is also annotated by experts. We provide a strategy for recruiting suitable non-experts. Our analysis of the inter-annotator agreements shows that the resulting annotations obtained from the majority vote of the non-experts are of comparable quality to the annotations of the experts. We provide analyses of the stance evolution in short term and long term levels, a comparison of language usage between users with vacillating and resolute attitudes, and fine-grained stance detection baselines.

A Comprehensive Survey of Detection and Prevention Approaches for Online Radicalization: Identifying Gaps and Future Directions

“…The changing of opinions over time on social networks was investigated by Sakketou et al [61]. The authors collected a dataset containing posts expressing the stance from different Reddit communities.…”

“…Theisen et al [60] used keywords like #echochambers, and #radicalization to collect thousands of user comments from Reddit threads belonging to extremist communities. [61] Proposed a SPINOS dataset to detect opinion shifts, they collected threads from both extremist and moderate communities from the Reddit platform using keywords like #gun control, #abortion, #veganism, and #politics in order to detect opinion shifts. Kennedy et al [62] proposed a dataset of hate speech from the Gab platform, collected over 27k posts.…”

“…Study Approach Dataset Type Mitigation Techniques 2023 [79] ML Imbalanced Na 2023 [101] Graph Imbalanced Na 2022 [21] ML,DL Imbalanced SMOTE 2022 [23] ML Balanced Na 2022 [22] ML,DL Balanced Na 2022 [61] ML Balanced Na 2022 [26] ML Imbalanced SMOTE 2022 [106] DL Imbalanced Na 2022 [28] DL Imbalanced weighted loss function 2022 [107] DL Balanced Na 2022 [108] DL Imbalanced oversampling, undersampling 2022 [110] Clustering Imbalanced SMOTE 2022 [114] Analysis Balanced Na 2022 [65] Analysis Balanced Na 2022 [98] Graph Balanced Na 2022 [19] Graph Imbalanced Na 2021 [60] ML Imbalanced undersampling 2021 [70] ML Imbalanced Na 2021 [85] DL Imbalanced oversampling, undersampling 2021 [79] DL Imbalanced oversampling, undersampling 2021 [115] Analysis Imbalanced weighted sampling 2021 [67] Analysis Imbalanced Na 2021 [118] Analysis Na Na 2021 [96] Graph Imbalanced Na 2021 [20] Graph Balanced Na 2021 [99] Graph Imbalanced oversampling 2020 [54] ML Imbalanced oversampling, undersampling 2020 [55] ML Imbalanced oversampling, undersampling 2020 [24] ML Imbalanced Na 2020 [91] ML Imbalanced SMOTE 2020 [58] ML Imbalanced SMOTE 2020 [81] ML Imbalanced SMOTE 2020 [27] ML Balanced Na 2020 [116] Analysis Imbalanced SMOTE 2020 [57] ML Imbalanced oversampling , undersampling 2019 [59] ML Imbalanced oversampling , undersampling 2019 [63] ML Imbal...…”

OK Boomer: Probing the socio-demographic Divide in Echo Chambers

“…As the opinion of the users towards the investigated topics is a central dimension when studying the echo chamber phenomenon, we discarded our automated stance classification efforts (F1-score around 60% on three classes) and utilized the human labels of the SPINOS dataset instead (Sakketou et al, 2022) 1 . Since the SPINOS dataset resembles a proper subset of the data that will be studied in this paper, it was deemed a feasible source for human labeled user stance samples.…”