Transformer Networks of Human Conceptual Knowledge

Abstract

We present a computational model capable of simulating aspects of human knowledge for thousands of real-world concepts. Our approach involves fine-tuning a transformer network for natural language understanding on participant-generated feature norms. We show that such a model can successfully extrapolate from its training dataset, and predict human knowledge for novel concepts and features. We also apply our model to stimuli from twenty-three previous experiments in semantic cognition research, and show that it reproduces fifteen classic findings involving semantic verification, concept typicality, feature distribution, and semantic similarity. We interpret these findings using established properties of classic connectionist networks. The success of our approach shows how the combination of natural language data and psychological data can be used to build cognitive models with rich world knowledge. Such models can be used in the service of new psychological applications, such as the cognitive process modeling of naturalistic semantic verification and knowledge retrieval, as well as the modeling of real-world categorization, decision making, and reasoning.

Human-like property induction is a challenge for large language models

“…As Figure 2C shows, the GPT-3 similarity ratings are correlated to some extent with human ratings. This is consistent with previous work suggesting that the internal representations of TLMs can be used to make reasonable predictions about human similarity judgments (Bhatia & Richie, 2021). GPT-3 accounts for some superordinate categories better than others, with correlations ranging between 0.16 (fish) and 0.58 (professions).…”

“…For psychologists, property induction is relevant to a literature that assesses TLMs and predecessors such as LSA (Landauer & Dumais, 1997) as computational accounts of the acquisition, use, and representation of semantic knowledge. Recent work has evaluated the extent to which TLMs account for human similarity ratings, typicality ratings, and response times (Bhatia & Richie, 2021;Lake & Murphy, 2021), but there has been relatively little work on inductive reasoning. A notable exception is the work of Misra, Ettinger, and Taylor Rayz (2021), who focus on typicality and include property induction as one of the tasks that they consider.…”

Probabilistic analogical mapping with semantic relation networks.

“…As the first step toward automating analogical mapping, we adopt semantic representations of individual words generated by a machine-learning model, Word2vec (Mikolov et al, 2013). Word2vec and similar models based on distributional semantics, such as Global Vectors (GloVe; Pennington et al, 2014) and Bidirectional Encoder Representations from Transformers (BERT; Devlin et al, 2019), have proved successful in predicting behavioral judgments of lexical similarity or association (Hill et al, 2015; Hofmann et al, 2018; Pereira et al, 2016; Richie & Bhatia, 2021), neural responses to word and relation meanings (Huth et al, 2016; Pereira et al, 2018; Zhang et al, 2020), and high-level inferences including assessments of probability (Bhatia, 2017; Bhatia et al, 2019) and semantic verification (Bhatia & Richie, in press). In the simulations reported here, the semantic meanings of individual concepts are represented by 300-dimensional embeddings created by Word2vec after training on a corpus of articles drawn from Google News.…”

Can Transformer Language Models Predict Psychometric Properties?

“…Common optimization tasks for pretraining transformers, such as the masked LM task (Devlin et al, 2018) are quite similar to the word prediction tasks that are known to predict children's performance on other linguistic skills (Borovsky et al, 2012;Neuman et al, 2011;Gambi et al, 2020). Finally, TLMs tend to outperform other LMs in recent work modeling human reading times, eye-tracking data, and other psychological and psycholinguistic phenomena (Merkx and Frank, 2021;Schrimpf et al, 2020b,a;Hao et al, 2020;Bhatia and Richie, 2020;.…”