The Problem of Semantic Shift in Longitudinal Monitoring of Social Media

Abstract

Social media allows researchers to track societal and cultural changes over time based on language analysis tools. Many of these tools rely on statistical algorithms which need to be tuned to specific types of language. Recent studies have shown the absence of appropriate tuning, specifically in the presence of semantic shift, can hinder robustness of the underlying methods. However, little is known about the practical effect this sensitivity may have on downstream longitudinal analyses. We explore this gap in the literature through a timely case study: understanding shifts in depression during the course of the COVID-19 pandemic. We find that inclusion of only a small number of semantically-unstable features can promote significant changes in longitudinal estimates of our target outcome. At the same time, we demonstrate that a recently-introduced method for measuring semantic shift may be used to proactively identify failure points of language-based models and, in turn, improve predictive generalization.

Leveraging Prompt-Based Large Language Models: Predicting Pandemic Health Decisions and Outcomes Through Social Media Language

“…The COVID-19 pandemic has ignited an unprecedented increase in social media discourse on health decisions and practices [113,122], spurring a wave of computational social science research [39,104] aimed at understanding this phenomenon in the field of HCI [63,77] and CSCW [16]. Using text mining and computational linguistics, researchers have analyzed pandemic-related social media discourse through the lens of mental health [75], political views [17,90], attitudes towards vaccines [79,119], misinformation [49,73,99],…”

“…For example, researchers have examined collective shifts in the public mood in response to the evolving pandemic news cycles by analyzing the daily sentiment of tweets [105]. Similarly, others have analyzed social media posts containing a subset of depression-indicative n-grams to track the fluctuation in mental health of social media users over the course of the pandemic [39].…”

Then and Now: Quantifying the Longitudinal Validity of Self-Disclosed Depression Diagnoses

“…Failure to comprehensively understand biases that arise under this methodology can have severe detrimental effects in downstream systems. In the case of estimating populationlevel health trends, for instance, we have already seen machine learning classifiers produce outcomes that are inconsistent across computational studies (Wolohan, 2020;Biester et al, 2021;Harrigian and Dredze, 2022) and in conflict with traditional measurement techniques (Amir et al, 2019). Continuing to pursue this line of research without questioning the validity of its underlying data has the potential to irreparably damage the public's trust in this domain, and worse, enable ill-informed decision making in highly-sensitive circumstances.…”