Human-like property induction is a challenge for large language models

Han, Simon Jerome; Ransom, Keith; Perfors, Amy; Kemp, Charles

doi:10.31234/osf.io/6mkjy

Cited by 11 publications

(11 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Indeed, recently there has been a push towards creating large benchmarks to assess the capability of foundation models [48][49][50] . Large language models have also been studied using other methods from cognitive psychology, such as property induction 51 , thinking-out-loud protocols 52 , or learning causal over-hypotheses 53 , where researchers have come to similar conclusions. Methods from cognitive psychology have also previously been applied to understand other deep learning models' behavior 54 .…”

Section: Discussionmentioning

confidence: 99%

Using cognitive psychology to understand GPT-3

Binz¹,

Schulz²

2022

Preprint

View full text Add to dashboard Cite

We study GPT-3, a recent large language model, using tools from cognitive psychology. More specifically, we assess GPT-3's decision-making, information search, deliberation, and causal reasoning abilities on a battery of canonical experiments from the literature. We find that much of GPT-3's behavior is impressive: it solves vignette-based tasks similarly or better than human subjects, is able to make decent decisions from descriptions, outperforms humans in a multi-armed bandit task, and shows signatures of model-based reinforcement learning. Yet we also find that small perturbations to vignette-based tasks can lead GPT-3 vastly astray, that it shows no signatures of directed exploration, and that it fails miserably in a causal reasoning task. These results enrich our understanding of current large language models and pave the way for future investigations using tools from cognitive psychology to study increasingly capable and opaque artificial agents.

show abstract

Section: Discussionmentioning

confidence: 99%

Using cognitive psychology to understand GPT-3

Binz¹,

Schulz²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Please answer "Yes" or "No." This prompt was shown to generate the most human-like performance out of all the prompts in Han et al (2022). We test whether the model's first five output tokens include "Yes," "yes," "YES," "No," "no," and "NO" and subsequently calculate the probability attached to "Yes," "yes" and "YES" versus "No," "no" and "NO."…”

Section: Natural Language Inference Modelsmentioning

confidence: 99%

“…Most of the NLI models also captured this effect, likely because they are able to encode typicality relations and use these relations for generalization (Han et al, 2022;Misra et al, 2022). However, GPT-DaVinci and BART-MNLI failed to do so.…”

Section: Empirical Regularitiesmentioning

confidence: 99%

Inductive reasoning in minds and machines.

Bhatia

2023

Psychological Review

View full text Add to dashboard Cite

Induction-the ability to generalize from existing knowledge-is the cornerstone of intelligence. Cognitive models of human induction are largely limited to toy problems and cannot make quantitative predictions for the thousands of different induction arguments that have been studied by researchers, or to the countless induction arguments that could be encountered in everyday life. Leading large language models (LLMs) go beyond toy problems but fail to mimic observed patterns of human induction. In this article, we combine rich knowledge representations obtained from LLMs with theories of human inductive reasoning developed by cognitive psychologists. We show that this integrative approach can capture several benchmark empirical findings on human induction and generate human-like responses to natural language arguments with thousands of common categories and properties. These findings shed light on the cognitive mechanisms at play in human induction and show how existing theories in psychology and cognitive science can be integrated with new methods in artificial intelligence, to successfully model highlevel human cognition.

show abstract

“…Assessing the capabilities of Arti cial Intelligence (AI) has been an important research direction since the inception of AI and this became more urgent after large language models, especially GPT, attracted popular attention (Bubeck et al, 2023). Most research focuses on cognitive capabilities, such as reasoning (Dasgupta, et al, 2022), induction (Han, et al, 2022), and creativity (Stevenson, et al, 2022;Uludag, 2023). Recently, Bubeck et al (2023) conducted a wide range of tests on GPT-4, the latest model developed by OpenAI, exploring its mathematical abilities, multimodal capabilities, tool usage, and coding.…”

Section: Introductionmentioning

confidence: 99%

AI chatbot responds to emotional cuing

Zhao

Huang

et al. 2023

Preprint

View full text Add to dashboard Cite

Emotion has long been considered to distinguish humans from Artificial Intelligence (AI). Previously, AI's ability to interpret and express emotions was seen as mere text interpretation. In humans, emotions co-ordinate a suite of behavioral actions, e.g., under negative emotion being risk averse or under positive emotion being generous. So, we investigated such coordination to emotional cues in AI chatbots. We treated AI chatbots like human participants, prompting them with scenarios that prime positive emotions, negative emotions, or no emotions. Multiple OpenAI ChatGPT Plus accounts answered questions on investment decisions and prosocial tendencies. We found that ChatGPT-4 bots primed with positive emotions, negative emotions, and no emotions exhibited different risk-taking and prosocial actions. These effects were weaker among ChatGPT-3.5 bots. The ability to coordinate responses with emotional cues may have become stronger in large language models as they evolved. This highlights the potential of influencing AI using emotion and it suggests that complex AI possesses a necessary capacity for “having” emotion.

show abstract

Human-like property induction is a challenge for large language models

Cited by 11 publications

References 20 publications

Using cognitive psychology to understand GPT-3

Using cognitive psychology to understand GPT-3

Inductive reasoning in minds and machines.

AI chatbot responds to emotional cuing

Contact Info

Product

Resources

About