Modeling Semantic Encoding in a Common Neural Representational Space

Abstract

Encoding models for mapping voxelwise semantic tuning are typically estimated separately for each individual, limiting their generalizability. In the current report, we develop a method for estimating semantic encoding models that generalize across individuals. Functional MRI was used to measure brain responses while participants freely viewed a naturalistic audiovisual movie. Word embeddings capturing agent-, action-, object-, and scene-related semantic content were assigned to each imaging volume based on an annotation of the film. We constructed both conventional within-subject semantic encoding models and between-subject models where the model was trained on a subset of participants and validated on a left-out participant. Between-subject models were trained using cortical surface-based anatomical normalization or surface-based whole-cortex hyperalignment. We used hyperalignment to project group data into an individual’s unique anatomical space via a common representational space, thus leveraging a larger volume of data for out-of-sample prediction while preserving the individual’s fine-grained functional–anatomical idiosyncrasies. Our findings demonstrate that anatomical normalization degrades the spatial specificity of between-subject encoding models relative to within-subject models. Hyperalignment, on the other hand, recovers the spatial specificity of semantic tuning lost during anatomical normalization, and yields model performance exceeding that of within-subject models.

An fMRI dataset in response to “The Grand Budapest Hotel”, a socially-rich, naturalistic movie

“…Multivariate Pattern Analysis (MVPA) 11 , 12 , including Representational Similarity Analysis (RSA) 13 , 14 , can be used to investigate information in population responses embedded in patterns of brain activity. Voxelwise encoding models can be used with naturalistic stimuli to create predictive models of brain activity and quantify complex, multidimensional voxel tuning 15 – 17 . Across subjects, brain activity is highly similar in response to naturalistic stimuli 5 , and such brain responses can be used as a basis for functional alignment (e.g., Hyperalignment 1 , 7 , 18 ).…”

An fMRI dataset in response to “The Grand Budapest Hotel”, a socially-rich, naturalistic movie

“…Multivariate Pattern Analysis (MVPA) 11,12 , including Representational Similarity Analysis (RSA) 13,14 , can be used to investigate information in population responses embedded in patterns of brain activity. Voxelwise encoding models can be used with naturalistic stimuli to create predictive models of brain activity and quantify complex, multidimensional voxel tuning [15][16][17] . Across subjects, brain activity is highly similar in response to naturalistic stimuli 5 , and such brain responses can be used as a basis for functional alignment (e.g., Hyperalignment 1,7,18 ).…”

Leveraging shared connectivity to aggregate heterogeneous datasets into a common response space

“…We also evaluated how cSRM impacts model-based encoding and decoding for two stories (Fig. 3; Güçlü and van Gerven, 2017;Van Uden et al, 2018;Vodrahalli et al, 2017;Wen et al, 2018). To quantify the semantic content of the stories, we first used a semi-supervised forced-alignment algorithm (Yuan and Liberman, 2008) to extract time-stamped transcripts from each story stimulus (see Fig.…”

“…However, these models are often estimated independently per subject using large volumes of data (e.g., Huth et al, 2016), which poses problems of both scalability (in terms of data collection) and generalizability across subjects (cf. Güçlü and van Gerven, 2017;Van Uden et al, 2018;Vodrahalli et al, 2017). Here we used a simplistic semantic encoding model to explore how cSRM impacts model performance.…”