Tweet Classification without the Tweet: An Empirical Examination of User versus Document Attributes

Abstract

NLP naturally puts a primary focus on leveraging document language, occasionally considering user attributes as supplemental. However, as we tackle more social scientific tasks, it is possible user attributes might be of primary importance and the document supplemental. Here, we systematically investigate the predictive power of user-level features alone versus document-level features for document-level tasks. We first show user attributes can sometimes carry more task-related information than the document itself. For example, a tweet-level stance detection model using only 13 user-level attributes (i.e. features that did not depend on the specific tweet) was able to obtain a higher F1 than the topperforming SemEval participant. We then consider multiple tasks and a wider range of user attributes, showing the performance of strong document-only models can often be improved (as in stance, sentiment, and sarcasm) with user attributes, particularly benefiting tasks with stable "trait-like" outcomes (e.g. stance) most relative to frequently changing "statelike" outcomes (e.g. sentiment). These results not only support the growing work on integrating user factors into predictive systems, but that some of our NLP tasks might be better cast primarily as user-level (or human) tasks.

MeLT: Message-Level Transformer with Masked Document Representations as Pre-Training for Stance Detection

“…We also include the top participant from the shared task Zarrella and Marsh (2016) which uses a different F1 score as defined for the shared task, referred to here as SemEval F1 3 . Lastly, we compare our results to the approach of Lynn et al (2019), from whom we received the extended history dataset, which uses the labeled tweet and a list of accounts the author follows. However, they only report the weighted-F1 score for their best performing model.…”

“…In total we have 3,021 instances with a split of 1658 train, 418 dev, and 945 test across all targets. The original 2016 shared task had 4,100 instances, however due to accounts or messages being deleted over time, we were unable to replicate the complete original dataset and instead used the smaller version available from Lynn et al (2019)…”

Characterizing Social Spambots by their Human Traits

“…Our work is aligned with a growing set of methods to to embed language processing within the social and human contexts they are applied (Lynn et al, 2019). Most similar is the work on language generation or dialog agents (i.e.…”

Empirical Evaluation of Pre-trained Transformers for Human-Level NLP: The Role of Sample Size and Dimensionality

“…Such tasks present an interesting challenge for the NLP community to model the people behind the language rather than the language itself, and the social scientific community has begun to see success of such approaches as an alternative or supplement to standard psychological assessment techniques like questionnaires (Kern et al, 2016;Eichstaedt et al, 2018). Generally, such work is helping to embed NLP in a greater social and human context (Hovy and Spruit, 2016;Lynn et al, 2019).…”