Exploring the Representations of Individual Entities in the Brain Combining EEG and Distributional Semantics

Abstract

Semantic knowledge about individual entities (i.e., the referents of proper names such as Jacinta Ardern) is fine-grained, episodic, and strongly social in nature, when compared with knowledge about generic entities (the referents of common nouns such as politician). We investigate the semantic representations of individual entities in the brain; and for the first time we approach this question using both neural data, in the form of newly-acquired EEG data, and distributional models of word meaning, employing them to isolate semantic information regarding individual entities in the brain. We ran two sets of analyses. The first set of analyses is only concerned with the evoked responses to individual entities and their categories. We find that it is possible to classify them according to both their coarse and their fine-grained category at appropriate timepoints, but that it is hard to map representational information learned from individuals to their categories. In the second set of analyses, we learn to decode from evoked responses to distributional word vectors. These results indicate that such a mapping can be learnt successfully: this counts not only as a demonstration that representations of individuals can be discriminated in EEG responses, but also as a first brain-based validation of distributional semantic models as representations of individual entities. Finally, in-depth analyses of the decoder performance provide additional evidence that the referents of proper names and categories have little in common when it comes to their representation in the brain.

Modeling Brain Representations of Words' Concreteness in Context Using GPT‐2 and Human Ratings

“…Modification of the semantic representations through semantic composition is the raison d'être of contextualized language models, which represent each word as a complex, nonlinear function of the other words surrounding them. Furthermore, our pattern of results is consistent with recent results in the literature which clearly indicates that contextualized language models provide excellent fit with brain data (Anderson et al., 2021; Bruera & Poesio, 2022; Caucheteux & King, 2022; Goldstein et al., 2022; Jat et al., 2019; Sun, Wang, Zhang, & Zong, 2020; Schrimpf et al., 2021). Note, however, that these previous works only considered either simple concepts or longer, less controlled sentences, while our work focused on an intermediate unit—phrases—and involved much more stringent testing: we used a strictly controlled modulation of concreteness and very specific cases of verb‐noun composition.…”

“…Note that, each round of pairwise evaluations measures to what extent the model has learnt to generalize in the specific case of the categorical relation holding between the two test items (Bruera & Poesio, 2022; Chyzhyk, Varoquaux, Milham, & Thirion, 2022; Elangovan, He, & Verspoor, 2021; Grootswagers et al., 2017; Gorman & Bedrick, 2019; Lake & Baroni, 2018). Therefore, it is possible to examine the results of different categorical relations separately.…”

“…To extract the contextualized vectors to be used for encoding, we adapted the so-called procedure of "representation pooling," a methodology validated on benchmarks from computational linguistics in Bommasani et al (2020), Vulić et al (2020), and Apidianaki (2022) and, more specifically, in Bruera and & Poesio (2022), for brain data. The procedure is equal for both single-words ITGPT2 and phrase ITGPT2.…”

EEG Searchlight Decoding Reveals Person- and Place-specific Responses for Semantic Category and Familiarity

“…We expected, according to previous literature (Proverbio et al, 2001;Bruera & Poesio, 2022) , to be able to decode information about semantic categories for both people and places within the N400 time range, distributed over centrotemporal electrodes.…”

“…Since the range of semantic phenomena affecting the N400 is extremely broad, a unified theoretical account of the N400 as a marker of semantic processing is still missing (Rabovsky, Hansen, & McClelland, 2018;Delogu, Brouwer, & Crocker, 2019;Eddine, Brothers, & Kuperberg, 2022;Šoškić, Jovanović, Styles, Kappenman, & Ković, 2022) . However, crucially for our focus, a number of previous studies found that the time range of the N400 can be modulated by various types of aspects of semantic processing of individual entities -including the ones modulated in our experiment: categorical information (Bruera & Poesio, 2022) ; face familiarity, since a familiarity-induced counterpart of the N400, the FN400, has also been observed (Curran & Hancock, 2007) (although it's not clear whether the two are actually functionally identical or not (Voss & Federmeier, 2011;Bridger, Bader, Kriukova, Unger, & Mecklinger, 2012;Leynes, Bruett, Krizan, & Veloso, 2017)) ; and additionally, the difference between proper names and common nouns (Proverbio, Lilli, Semenza, & Zani, 2001;Proverbio, Mariani, Zani, & Adorni, 2009;Adorni, Manfredi, & Proverbio, 2014;Sulpizio & Job, 2018) .…”