A Systematic Evaluation of Text Mining Methods for Short Texts: Mapping Individuals’ Internal States from Online Posts

Abstract

Sociologists have successfully used text mining to investigate discourse using news articles, official documents, and other sources. Yet, the potential of exploring millions of short texts generated spontaneously by individuals in online environments has remained untapped within the field. To fill this gap, we show how such texts can inform sociologists about individual internal states such as norms, motives, and stances, which thus far have been mainly elicited using surveys. We assess the performance of 581 variations of three text mining approaches–dictionary methods, supervised, and unsupervised machine learning–against the benchmark of texts coded by humans for complex schemes capturing individuals’ internal states. Our analysis includes coding feedback texts from an online market for motives for leaving feedback (N = 2,000) and tweet texts for moral values expressed in text (N = 3,832). We describe challenges arising with these different approaches and provide best-practice advice for future applications.

Text mining for social science – The state and the future of computational text analysis in sociology

“…3 Despite the rich tradition of quantitative computer-assisted text analysis, the recent breakthrough in the use of text mining in sociological research mostly came from a more qualitative tradition (e.g., the 2013 issue of journal Poetics). Text mining has since been used to support (or replicate) diverse manual approaches to text analysis, ranging from quantitative network analysis (Goldenstein and Poschmann 2019b;Sudhahar et al, 2015a) and systematic manual coding (Macanovic and Przepiorka 2022;Rona-Tas et al, 2019) to qualitative hermeneutics Mohr et al 2015) and grounded theory (Muller et al, 2016;Nelson 2017). Most importantly, however, computational approaches allow for both quantification and nuanced reading of texts (Wiedemann 2016), closing the gap between quantitative and qualitative approaches to text analysis (DiMaggio et al, 2013;Wiedemann 2016).…”

“…These methods infer word patterns that characterize different coding categories (Nelson et al, 2018) and then use these patterns to code new, previously unseen texts. While such algorithms replicate trained human coding using complex schemes rather well (Hoover et al, 2020;Macanovic and Przepiorka 2022;Nelson et al, 2018), they are currently only scarcely used in the field. Macanovic and Przepiorka (2021) manually code 2000 texts to infer motives that drive buyers in online markets to write feedback about their transactions.…”

“…While such algorithms replicate trained human coding using complex schemes rather well (Hoover et al, 2020;Macanovic and Przepiorka 2022;Nelson et al, 2018), they are currently only scarcely used in the field. Macanovic and Przepiorka (2021) manually code 2000 texts to infer motives that drive buyers in online markets to write feedback about their transactions. Using supervised machine learning, authors are able to expand this manual coding onto 2 million feedback texts and granularly explore the role of different motives in generating feedback after various market experiences.…”

“…Using supervised machine learning, authors are able to expand this manual coding onto 2 million feedback texts and granularly explore the role of different motives in generating feedback after various market experiences. Their analysis shows how other-regarding motives present an integral element of the successful functioning of large-scale anonymous online marketplaces (Macanovic and Przepiorka 2021). Rona-Tas et al ( 2019) use supervised methods to code complex food safety risk assessments for the level of scientific uncertainty they express.…”

“…Sociologists have used novel computational methods to explore a variety of phenomena, including those clearly related to textual expression: the discourse of and surrounding the political elites (van Atteveldt et al, 2008;Bonikowski and Gidron 2016;DiMaggio et al 2013), the rhetoric of online communities (Davidson et al, 2017;Törnberg 2016a, 2016b) and social groups and movements (Almquist and Bagozzi 2019;Nelson 2021), the varieties of event framing in media (Bastin and Bouchet-Valat 2014;Franzosi et al 2012), and the nature of public debates surrounding different societal issues (Bail et al 2017). Computational text analysis has supported both research concerned with the evolution of cultural meanings at a macro level (Boutyline et al 2020;Kozlowski et al, 2019) and inference on values and cultural schemas at the micro level (Macanovic and Przepiorka 2021;Taylor and Stoltz 2020). Finally, computational methods have helped us understand the inner workings of science (McMahan and Evans 2018;Rona-Tas et al, 2019;Schwemmer and Wieczorek 2020) and supported new methodological tools (Nardulli et al 2015;Zhang and Pan 2019).…”