ConStance: Modeling Annotation Contexts to Improve Stance Classification

Abstract

Manual annotations are a prerequisite for many applications of machine learning. However, weaknesses in the annotation process itself are easy to overlook. In particular, scholars often choose what information to give to annotators without examining these decisions empirically. For subjective tasks such as sentiment analysis, sarcasm, and stance detection, such choices can impact results. Here, for the task of political stance detection on Twitter, we show that providing too little context can result in noisy and uncertain annotations, whereas providing too strong a context may cause it to outweigh other signals. To characterize and reduce these biases, we develop ConStance, a general model for reasoning about annotations across information conditions. Given conflicting labels produced by multiple annotators seeing the same instances with different contexts, ConStance simultaneously estimates gold standard labels and also learns a classifier for new instances. We show that the classifier learned by ConStance outperforms a variety of baselines at predicting political stance, while the model's interpretable parameters shed light on the effects of each context.

Muse-ing on the Impact of Utterance Ordering on Crowdsourced Emotion Annotations

“…Hypothesis: Individual annotators differ in annotation similarity in the contextual presentations, compared to the randomized presentation. Joseph et al in [25] show that while insufficient context results in noisy and uncertain annotations, an overabundance of context may cause the context to outweigh other signals and lead to lower agreement. Further, contextual information biases different people differently on both temporal and intensity metrics [26,27].…”

Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries

“…– Manual annotation may yield subjective and noisy labels . Many factors affect the quality of human-annotations, including: (i) unreliable annotators, (ii) poorly specified annotation tasks and guidelines, (iii) poor category design (categories that are too broad, too narrow, or too vague), or (iv) insufficient information to make a reliable assessment (Cheng and Cosley, 2013 ; Joseph et al, 2017 ). Though the goal of an assessment task is to provide human input, underspecification or appeal to subjective judgment can introduce unintended biases that are often hard to detect.…”

Tweet Classification without the Tweet: An Empirical Examination of User versus Document Attributes

“…Recent work has shown that considering language within the context of user attributes can improve classification accuracy (Volkova et al, 2013;Bamman et al, 2014;Yang and Eisenstein, 2015;Hovy, 2015;Kulkarni et al, 2016;Lynn et al, 2017). Other work has used network or other meta data, such as in Bamman and Smith (2015); Johnson and Goldwasser (2016); Joseph et al (2017); Khattri et al (2015). In a sense these trail-blazing works might be viewed as case studies on user attributes -identifying particular pieces of information for particular tasks where user information has lead to an advantage.…”