Narrative Detection and Feature Analysis in Online Health Communities

Abstract

Narratives have been shown to be an effective way to communicate health risks and promote health behavior change, and given the growing amount of health information being shared on social media, it is crucial to study healthrelated narratives in social media. However, expert identification of a large number of narrative texts is a time consuming process, and larger scale studies on the use of narratives may be enabled through automatic text classification approaches. Prior work has demonstrated that automatic narrative detection is possible, but modern deep learning approaches have not been used for this task in the domain of online health communities. Therefore, in this paper, we explore the use of deep learning methods to automatically classify the presence of narratives in social media posts, finding that they outperform previously proposed approaches. We also find that in many cases, these models generalize well across posts from different health organizations. Finally, in order to better understand the increase in performance achieved by deep learning models, we use feature analysis techniques to explore the features that most contribute to narrative detection for posts in online health communities.

Narrative Style and the Spread of Health Misinformation on Twitter

“…The results for all models on the test set are presented in Table 3. Our findings align with those of Ganti et al (2022), who found that transformer-based models consistently outperform bag-of-words models for narrative detection, with the fine-tuned base RoBERTa model performing best overall, outperforming even the version of RoBERTa pretrained specifically on Twitter data. The GPT-3 text-davinci-003 model achieved high precision but lacked in recall, resulting in an F1-score similar to the bag-of-words models, though it requires less than 1% of the amount of training data (in the form of examples for in-context learning).…”

“…While most studies of narratives have relied on manual annotations, Dirkson et al (2019) and Verberne et al ( 2019) used human-assigned labels to train text classification models to detect narrativity in social media posts in the health domain. Both studies focused on bag-of-words models with ngram features, while more recent research found that deep learning models were more successful than classical machine learning models at detecting narratives in Facebook posts (Ganti et al, 2022). These studies suggest that automatic narrative classification could enable the labeling of even larger social media datasets.…”