On the Reliability and Validity of Detecting Approval of Political Actors in Tweets

Abstract

Social media sites like Twitter possess the potential to complement surveys that measure political opinions and, more specifically, political actors' approval. However, new challenges related to the reliability and validity of social-media-based estimates arise. Various sentiment analysis and stance detection methods have been developed and used in previous research to measure users' political opinions based on their content on social media. In this work, we attempt to gauge the efficacy of untargeted sentiment, targeted sentiment, and stance detection methods in labeling various political actors' approval by benchmarking them across several datasets. We also contrast the performance of these pretrained methods that can be used in an off-the-shelf (OTS) manner against a set of models trained on minimal custom data. We find that OTS methods have low generalizability on unseen and familiar targets, while low-resource custom models are more robust. Our work sheds light on the strengths and limitations of existing methods proposed for understanding politicians' approval from tweets.

Beyond facts – a survey and conceptualisation of claims in online discourse analysis

“…Similarly, the process of stance detection is challenging as it has been shown to not work well for the minority class, i.e. documents disagreeing with the claim [70,154], and for unseen targets [160]. Little research in viewpoint discovery deals with extracting viewpoints for more than two polarized positions, a topic that could be worthwhile researching for the analysis of debates.…”

“…In recent work, Sen et al [160] compare untargeted and targeted opinion mining methods (sentiment analysis, aspect-based sentiment analysis, stance detection) to infer approval of political actors in tweets. They show that the compared targeted approaches have low generalizability on unseen and unfamiliar targets and that indirectly expressed stances are hard to detect, and thus identify the need for further research in this area.…”

(Mis)alignment Between Stance Expressed in Social Media Data and Public Opinion Surveys

“…While early efforts to use social media to extract public opinion focused on sentiment analysis (O'Connor et al, 2010;Mitchell et al, 2013;Conrad et al, 2019), more recent work has shifted towards stance detection, which more directly aligns with the goal of public opinion modeling (Sen et al, 2020;Mohammad et al, 2017). In stance detection, the task moves from estimating the positive or negative sentiment of a given text to evaluating whether the authoring individual is for, against, or neutral towards some target concept.…”

Perspective: Acknowledging Data Work in the Social Media Research Lifecycle

“…In a more fine-grained view we saw that it is a challenge during this phase to untangle how single choices in the data acquisition process may have influenced the validity of the data. Deciding upon the exact selection criteria (e.g., search terms used to retrieve single posts, the chosen time period for data collection, the focus on certain languages) may indicate a specific focus, limit the scope of the research, or induce errors if the aim is to infer knowledge about whole populations ( Olteanu et al, 2019 ; Sen et al, 2019 ). However, the level of detail of documentation required to understand and reconstruct the data acquisition process from the perspective of a (reviewing) peer or secondary user goes beyond what is feasible to include in the “methods” section of a paper (see Zimmer and Proferes, 2014 or Hemphill et al, 2019 ).…”

“…These may need to be improved or adapted, as it is not always possible or feasible to design own tools. Existing methods and tools for data preparation and analysis may have certain limitations, e.g., tools for detecting sentiments or opinions in texts or users' gender and age based on profile photos are limited in accuracy - while the exact performance in a specific use case may be difficult to assess [see e.g., Sen et al (2020) for a comparison of different opinion mining approaches].…”