Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures

Abstract

Topic models, as developed in computer science, are effective tools for exploring and summarizing large document collections. When applied in social science research, however, they are commonly used for measurement, a task that requires careful validation to ensure that the model outputs actually capture the desired concept of interest. In this paper, we review current practices for topic validation in the field and show that extensive model validation is increasingly rare, or at least not systematically reported in papers and appendices. To supplement current practices, we refine an existing crowd-sourcing method by Chang and coauthors for validating topic quality and go on to create new procedures for validating conceptual labels provided by the researcher. We illustrate our method with an analysis of Facebook posts by U.S. Senators and provide software and guidance for researchers wishing to validate their own topic models. While tailored, case-specific validation exercises will always be best, we aim to improve standard practices by providing a general-purpose tool to validate topics as measures.

A Systematic Evaluation of Text Mining Methods for Short Texts: Mapping Individuals’ Internal States from Online Posts

“…UML methods cluster data without relying on manual coding, finding similar words or texts in the data and grouping them together . While they are best suited for exploratory analyses, these methods can be repurposed to measurement of concepts of interest in text Ying, Montgomery, and Stewart 2022). Here we assess their performance in capturing subtle internal states in short texts using the following procedure: (1) we use UML models to group texts, (2) examine whether some of these groups correspond to our coding categories and, if so, (3) code texts that belong to these groups for corresponding categories.…”

“…Here we assess their performance in capturing subtle internal states in short texts using the following procedure: (1) we use UML models to group texts, (2) examine whether some of these groups correspond to our coding categories and, if so, (3) code texts that belong to these groups for corresponding categories. While we rely on our interpretation and expertise in matching groups to coding categories, researchers can resort to more sophisticated methods for the evaluation, validation, and labelling of results obtained with UML models (Ying et al 2022). In this overview, we survey several UML algorithms.…”

Twitter data from the 2019–20 Australian bushfires reveals participatory and temporal variations in social media use for disaster recovery

“…The researchers appreciate that there are existing topic modelling techniques and machine learning approaches for computationally extracting topics and classifying users 34 – 36 . However, the researchers have gone through the pain of manually annotating the data because of our research interest to further develop knowledge about the identified topics and user categories.…”

Researcher reasoning meets computational capacity: Machine learning for social science

“…This path forward is promising but also entails risks, because there is no guarantee that an algorithm operating on word frequencies will arrive at a meaningful definition of a cultural category. For this reason, unsupervised methods to summarize text data always place the burden on the researcher to justify their chosen interpretation and validate the utility of the topics learned (Grimmer et al, 2022;Grimmer and Stewart, 2013;Ying et al, 2021). LDA thus illustrates a key idea that applies more broadly to unsupervised methods: while these methods may appear to inductively discover insights from the data alone, they actually involve extensive theoretical work on the part of the researcher to justify and interpret the result.…”