What Sounds “Right” to Me? Experiential Factors in the Perception of Political Ideology

Abstract

In this paper, we challenge the assumption that political ideology is inherently built into text by presenting an investigation into the impact of experiential factors on annotator perceptions of political ideology. We construct an annotated corpus of U.S. political discussion, where in addition to ideology labels for texts, annotators provide information about their political affiliation, exposure to political news, and familiarity with the source domain of discussion, Reddit. We investigate the variability in ideology judgments across annotators, finding evidence that these experiential factors may influence the consistency of how political ideologies are perceived. Finally, we present evidence that understanding how humans perceive and interpret ideology from texts remains a challenging task for state-ofthe-art language models, pointing towards potential issues when modeling user experiences that may require more contextual knowledge.

(Mis)alignment Between Stance Expressed in Social Media Data and Public Opinion Surveys

“…Joseph et al (2017) showed that how one constructs annotation tasks can significantly impact (supervised) model performance and one's assessment of it. Further, as demonstrated by Shen and Rose (2021) on the closely related task of inferring political ideology, annotator expertise and subjectivity also play an important role in the quality of annotated data.…”

“…The present work complements these prior efforts by delving into other questions of annotator disagreement and inference. Whereas prior work has considered disagreement arising from task differences (Joseph et al, 2017), or properties of the annotators (Shen and Rose, 2021), we control for both of these factors, taking a single task and a relatively homogenous set of expert annotators. Instead, extending recent work studying prediction on multiple targets (van den Berg et al, 2019;, we study how agreement varies depending on the target selected, and how even within a single task design, annotators can come to rely on distinct subsets of information.…”

A study towards contextual understanding of toxicity in online conversations

“…However, the contextual information is local and does not take into account global subject content under discussion. Shen and Rose (2021) investigate the influence of annotators’ political beliefs and familiarity of the source towards annotations of identifying political orientation of Reddit posts. Chung et al (2019) present a dataset that collects both hateful and toxic messages along with potential repairs that can provide counter-narrative information with fact-based information and non-offensive language to de-escalate hateful discourse.…”

Proceedings of the Second Workshop on Bridging Human--Computer Interaction and Natural Language Processing

“…Our findings from simulations provide directions for user experiments. Human perception -and thus human annotators' interpretation -is influenced by human factors such as preferences, cultural differences, bias, domain expertise, fatigue, time on task, or mood at annotation time (Alm, 2012;Amidei et al, 2020;Shen and Rose, 2021). Generally, experts with long-standing practice or in-depth knowledge may also not share consensus (Plank et al, 2014).…”