Dynamical Integrity and Its Background

Lenci, Stefano

doi:10.1007/978-3-031-45101-0_11

Cited by 1 publication

(2 citation statements)

References 90 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We, therefore, speculate that the acquisition of robust, statistics‐invariant world knowledge representations would require a different objective function from that required for acquiring linguistic proficiency (Mahowald et al., 2023). The word‐in‐context prediction objective, which enables LLMs to excel at acquiring formal linguistic competence, encourages pretrained LLMs to organize their semantic spaces mainly by relatively simple features such as similarity and association (Lenci, 2023). This organization principle, however, does not always lead to robust concepts and relations, which are useful for natural language understanding tasks and serve as important units for developing more complex semantic structures (Lenci, 2023; Lenci & Sahlgren, 2023).…”

Section: Discussionmentioning

confidence: 99%

“…To assess the plausibility of an arbitrary event, a successful model of GEK must, therefore, acquire robust, generalizable representations of a vast number of actions and their associated restrictions on event participants. Many traditional and current distributional models have been argued to lack the representations of these building blocks for more complex semantic structures (Lenci, 2023; Lenci & Sahlgren, 2023; Pedinotti et al., 2021; Zhu, Li, & De Melo, 2018). The acquisition of GEK is complicated even more because the frequency with which events are reported in the pragmatically influenced texts available in the world is not a robust indicator of the frequency with which they occur in the real world (Gordon & Van Durme, 2013; see also Section 4.3).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Event Knowledge in Large Language Models: The Gap Between the Impossible and the Unlikely

Kauf,

Ivanova,

Rambelli

et al. 2023

Cognitive Science

View full text Add to dashboard Cite

Word co‐occurrence patterns in language corpora contain a surprising amount of conceptual knowledge. Large language models (LLMs), trained to predict words in context, leverage these patterns to achieve impressive performance on diverse semantic tasks requiring world knowledge. An important but understudied question about LLMs’ semantic abilities is whether they acquire generalized knowledge of common events. Here, we test whether five pretrained LLMs (from 2018's BERT to 2023's MPT) assign a higher likelihood to plausible descriptions of agent−patient interactions than to minimally different implausible versions of the same event. Using three curated sets of minimal sentence pairs (total n = 1215), we found that pretrained LLMs possess substantial event knowledge, outperforming other distributional language models. In particular, they almost always assign a higher likelihood to possible versus impossible events (The teacher bought the laptop vs. The laptop bought the teacher). However, LLMs show less consistent preferences for likely versus unlikely events (The nanny tutored the boy vs. The boy tutored the nanny). In follow‐up analyses, we show that (i) LLM scores are driven by both plausibility and surface‐level sentence features, (ii) LLM scores generalize well across syntactic variants (active vs. passive constructions) but less well across semantic variants (synonymous sentences), (iii) some LLM errors mirror human judgment ambiguity, and (iv) sentence plausibility serves as an organizing dimension in internal LLM representations. Overall, our results show that important aspects of event knowledge naturally emerge from distributional linguistic patterns, but also highlight a gap between representations of possible/impossible and likely/unlikely events.

show abstract

Section: Discussionmentioning

confidence: 99%