Expert-Informed Topic Models for Document Set Discovery

Abstract

The first step in many text-as-data studies is to find documents that address a specific topic within a larger document set. Researchers often rely on simple keyword searches to do this, even though this may introduce considerable selection bias. Such bias may be even greater when researchers lack the domain knowledge required to make informed search decisions, for example, in cross-national research or research on unfamiliar social contexts. We propose expert-informed topic modeling (EITM) as a hybrid approach to tackle this problem. EITM combines the validity of external domain knowledge captured through expert surveys with probabilistic topic models to help researchers identify subsets of documents that cover initially unknown domain-specific topics, such as specific events and debates, that belong to a researcher-defined master topic. EITM is a flexible and efficient approach to the thematic selection of documents from large text corpora for further study. We benchmark and validate the method by discovering blog posts that address the public role of religion within large corpora of Australian, Swiss, and Turkish blog posts and provide researchers with a complete workflow to guide the application of EITM in their own work.

Big textual data research for operations management: topic modelling with grounded theory

“…The main approach in these studies is reliant on the researchers' secondary data analysis. Although two previous studies (Kaplan and Vakili, 2015;Rinke et al, 2021) engaged with experts, their involvement was limited to providing labels and semantic ratings for their study. In contrast, we contributed to these methods by seeking insights from participants by asking their reasoning for enhanced analysis CGT.…”

Is Constructive Engagement Online a Lost Cause? Toxic Outrage in Online User Comments Across Democratic Political Systems and Discussion Arenas

“…Likewise, to build this corpus, an expert survey was conducted among 76 communication and religious studies scholars in the societies, who named relevant debates on the public role of religion and secularism in the respective society and a list of keywords associated with each debate. Based on these keywords, the articles and blog posts were selected in a novel, expert-informed topic modeling process (for detailed information see Rinke et al, 2021). The base corpus and the expert survey results then guided the following four data collection paths:…”

“…In total, 76 Facebook pages of partisan collective actors and 41 Facebook pages of alternative media were selected for analysis (Supplemental Appendix B/B-3). We collected all entries posted by these pages in the period of investigation and scored them for subject relevance with topic models that were built from extensive text corpora (Rinke et al, 2021) and that relied on the expert survey keywords. A cut-off for relevance was defined with gold standards of n = 300 comments in each country, each of which was scored by two trained coders with Krippendorff’s α nominal of .78.…”

Unpacking the Determinants of Outrage and Recognition in Public Discourse: Insights Across Socio-Cultural Divides, Political Systems, and Media Types

“…In order to draw a representative sample from the Mannheim International News Discourse Data Set (MIND; Rinke et al 2019) for manual content analysis we used the approach of expert-informed topic modeling (EITM; Rinke et al 2021) to identify the population of relevant articles available in a text format. EITM is an efficient approach combining expert domain knowledge and automated classification algorithms to identify and rank articles belonging to a specific master topic in unstructured text corpora.…”