A naturalistic neuroimaging database for understanding the brain using ecological stimuli

Abstract

Neuroimaging has advanced our understanding of human psychology using reductionist stimuli that often do not resemble information the brain naturally encounters. It has improved our understanding of the network organization of the brain mostly through analyses of ‘resting-state’ data for which the functions of networks cannot be verifiably labelled. We make a ‘Naturalistic Neuroimaging Database’ (NNDb v1.0) publically available to allow for a more complete understanding of the brain under more ecological conditions during which networks can be labelled. Eighty-six participants underwent behavioural testing and watched one of 10 full-length movies while functional magnetic resonance imaging was acquired. Resulting timeseries data are shown to be of high quality, with good signal-to-noise ratio, few outliers and low movement. Data-driven functional analyses provide further evidence of data quality. They also demonstrate accurate timeseries/movie alignment and how movie annotations might be used to label networks. The NNDb can be used to answer questions previously unaddressed with standard neuroimaging approaches, progressing our knowledge of how the brain works in the real world.

Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing

“…While we largely replicated these results in the Sherlock data, they did not generalize to the Summer data. This discrepancy may be related to the substantially lower signal-to-noise ratio in Summer movie data (maximum ISC value 0.73 vs. 0.56, see inter-subject brain correlation in Methods) perhaps due to the fact that the data were collected on a 1.5 T MRI scanner (Aliko et al, 2020). Another important distinction is how we annotated the theory of mind feature.…”

“…In the first study (Chen et al, 2017), 17 participants watched the first episode of the Sherlock BBC TV series (duration ~ 45 mins) in the scanner. In the second study (Aliko et al, 2020), 86 participants watched 10 different movies from 10 genres. We selected 20 participants who watched a commercial movie, 500 Days of Summer (duration ~ 90 mins).…”

“…Timing correction was also applied to align the fMRI timeseries and the movie. Detailed information on how movie was paused and restarted for each run and timing correction related this issue can be found in the original study (Aliko et al, 2020). In addition, similar to the Sherlock study, we shifted Summer fMRI by 4 s (4 TRs) from the stimulus onset to account for the hemodynamic delay.…”

“…Here, we implemented computer vision techniques, machine-learning-based encoding model analyses, and variance partitioning to identify the unique contribution of social interactions to responses in the human brain. To improve ecological generalizability, we performed the same analyses on two different fMRI datasets (Aliko et al, 2020; Chen et al, 2017) collected by different labs showing different movies from different genres (crime vs. romance) to different groups of participants living in different countries (US vs. UK) on different MRI scanners (3T vs. 1.5T). Our findings reveal that that social-affective features, consisting of an agent speaking, social interactions, theory of mind, perceived valence, and arousal, independently contribute to predicting brain responses in the STS and the mPFC.…”

A studyforrest extension, an annotation of spoken language in the German dubbed movie “Forrest Gump” and its audio-description

“…Naturalistic stimuli are usually designed for commercial purposes and to entertain their audiences. Thus, the temporal structure of their feature space is usually not explicitly known, leading to an “annotation bottleneck” 6 when used for neuroscientific research.…”