Predicting High-Level Human Judgment Across Diverse Behavioral Domains

Abstract

Recent advances in machine learning, combined with the increased availability of large natural language datasets, have made it possible to uncover semantic representations that characterize what people know about and associate with a wide range of objects and concepts. In this paper, we examine the power of word embeddings, a popular approach for uncovering semantic representations, for studying high-level human judgment. Word embeddings are typically applied to linguistic and semantic tasks, however we show that word embeddings can be used to predict complex theoretically-and practically-relevant human perceptions and evaluations in domains as diverse as social cognition, health behavior, risk perception, organizational behavior, and marketing. By learning mappings from word embeddings directly onto judgment ratings, we outperform a similarity-based baseline and perform favorably compared to common metrics of human inter-rater reliability. Word embeddings are also able to identify the concepts that are most associated with observed perceptions and evaluations, and can thus shed light on the psychological substrates of judgment. Overall, we provide new methods and insights for predicting and understanding high-level human judgment, with important applications across the social and behavioral sciences.

Semantic Norm Extrapolation is a Missing Data Problem

“…We admit that the best regularized regression method is unknown, but feel that LASSO regression is a fair comparison for improper single regression mean imputation methods (i.e. linear and ridge regressions in Gultchin et al, 2019;Hollis et al, 2017;Mandera et al, 2015;Recchia et al, 2015;Richie et al, 2019;Thompson et al, 2018;van Paridon et al, 2019;Westbury et al, 2013). Which type of regularized regression has the more desirable bias in this context is difficult to determine, due to the black-box nature of word vectors, and we leave it for future work.…”

“…The most common method for semantic norm extrapolation is linear regression, with a possible ridge penalty on the coefficients (Gultchin et al, 2019;Hollis et al, 2017;Mandera et al, 2015;Martıńez-Huertas et al, 2020;Paetzold et al, 2016;Recchia et al, 2015;Richie et al, 2019;Thompson et al, 2018;Turton et al, 2020;Utsumi, 2018;van Paridon et al, 2019;Westbury et al, 2013). Thus, we chose a form of linear regression as our stand-in in the simulations for existing methods of extrapolating semantic norms.…”

“…What supervised learning methods have been proposed for semantic norm extrapolation? Methods include linear regression (Gultchin, Patterson, Baym, Swinger, & Kalai, 2019;Hollis, Westbury, & Lefsrud, 2017;Mandera, Keuleers, & Brysbaert, 2015; Huertas, Jorge-Botana, Luzón, & Olmos, 2020;Thompson & Lupyan, 2018;Turton, Vinson, & Smith, 2020;Utsumi, 2018;Westbury, Shaoul, Hollis, Smithson, Briesemeister, Hofmann, & Jacobs, 2013), ridge regression (Paetzold & Specia, 2016;Recchia & Louwerse, 2015;Richie, Zou, & Bhatia, 2019;Turton et al, 2020;van Paridon et al, 2019), locally-weighted regression (Wang, Yu, Lai, & Zhang, 2016), random forests (Mandera et al, 2015;Turton et al, 2020), k-nearest neighbours (Bestgen, 2002;Bestgen & Vincze, 2012;Mandera et al, 2015;Martıńez-Huertas et al, 2020;Turton et al, 2020;Van Rensbergen, De Deyne, & Storms, 2016), exemplar based methods (Köper & Im Walde, 2016;Paetzold et al, 2016), feed-forward neural networks (Buechel, Rücker, & Hahn, 2020;Köper & im Walde, 2017;Martıńez-Huertas et al, 2020;Turton et al, 2020;Utsumi, 2018), support vector machines with a linear kernel (Li, Lu, Long, & Gui, 2017), support vector machines with a polynomial kernel (Paetzold et al, 2016), and support vector machines with a radial basis function kernel (Ljubešić, Fišer, & Peti-Stantić, 2018).…”

Transformer Networks of Human Conceptual Knowledge

“…However, even though distributed semantic representations predict the semantic relatedness of concepts with high accuracy and can be applied to tens of thousands of common concepts, they do not possess complex featural and relational representations for concepts. In response, researchers have begun to combine distributed semantic representations with survey and experimental data, such as feature norms and participant ratings (Andrews et al, 2009;Derby et al, 2019;Lu et al, 2019;Richie et al, 2019). In most such applications, distributed semantic representations for concepts serve as inputs into more complex models that are fine-tuned on participant data, and are thus able to proxy the (often structured) knowledge at play in the participants' responses.…”

Process and Content in Decisions from Memory

“…Such semantic representations describe the strength of association between judgment questions and response options, and can thus quantitatively model associative processes in probability judgment, factual judgment, social judgment, political judgment, and multiattribute choice (Bhatia, 2017b(Bhatia, , 2017cBhatia et al, 2018;Bhatia & Walasek, 2019;Caliskan et al, 2017;Garg et al, 2018;Hopkins, 2018;DECISIONS FROM MEMORY 58 Holtzman et al, 2011). By specifying what people know and associate with judgment and decision targets, vector semantic representations also allow researchers to build powerful predictive models that take semantic vectors as inputs and output a judgment or decision (Bhatia, in press;Bhatia & Stewart, 2018;Richie et al, 2019). In this paper we further illustrate the value of semantic vectors by showing how they can be used to model the content of thoughts during naturalistic decision making.…”