Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions

Abstract

Models that predict brain responses to stimuli provide one measure of understanding of a sensory system and have many potential applications in science and engineering. Deep artificial neural networks have emerged as the leading such predictive models of the visual system but are less explored in audition. Prior work provided examples of audio-trained neural networks that produced good predictions of auditory cortical fMRI responses and exhibited correspondence between model stages and brain regions, but left it unclear whether these results generalize to other neural network models and, thus, how to further improve models in this domain. We evaluated model-brain correspondence for publicly available audio neural network models along with in-house models trained on 4 different tasks. Most tested models outpredicted standard spectromporal filter-bank models of auditory cortex and exhibited systematic model-brain correspondence: Middle stages best predicted primary auditory cortex, while deep stages best predicted non-primary cortex. However, some state-of-the-art models produced substantially worse brain predictions. Models trained to recognize speech in background noise produced better brain predictions than models trained to recognize speech in quiet, potentially because hearing in noise imposes constraints on biological auditory representations. The training task influenced the prediction quality for specific cortical tuning properties, with best overall predictions resulting from models trained on multiple tasks. The results generally support the promise of deep neural networks as models of audition, though they also indicate that current models do not explain auditory cortical responses in their entirety.

From Cosine Similarity to Likelihood Ratio: Coupling Representations From Machine Learning (and Other Sources) With Cognitive Models

“…What all of these representation types have in common is that similarity relations between representations are characterized in terms of cosines. This applies to a large variety of machine learning models, not just those used for words and still images, but also dynamic stimuli like audio (Kell et al, 2018;Tuckute et al, 2023) and video (Lotter et al, 2017). C2L thus enables cognitive models to be applied to the increasingly complex and naturalistic items that machine learning models will be able to process.…”

Reconstructing Voice Identity from Noninvasive Auditory Cortex Recordings

“…In recent years, deep neural networks (DNNs) have emerged as a powerful tool for representing complex visual data, such as images (LeCun et al, 2015) or videos (Liu et al, 2020). In the auditory domain, DNNs have been shown to provide valuable representations-so-called feature or latent spaces-for modeling the cerebral processing of sound (brain encoding) (speech: Kell et al, 2018;Millet et al, 2022;Tuckute & Feather, 2023; semantic content: Caucheteux et 3 al., 2023;Giordano et al, 2023;music: Güçlü et al, 2016), or reconstructing the stimuli listened by a participant (brain decoding) (Akbari et al, 2019). They have not yet been used to explain cerebral representations of identity-related information due in part to the focus on speech information (von Kriegstein et al, 2003).…”

“…We addressed this question by using representational similarity analysis (RSA; Kriegeskorte et al, 2008) to test which model better accounts for the representational geometry for voice identities in the auditory cortex. Using RSA as a model comparison framework is relevant to examining the brain-model relationship from complementary angles (Diedrichsen & Kriegeskorte, 2017;Giordano et al, 2023;Tuckute & Feather, 2023). We built speaker x speaker representational dissimilarity matrices (RDMs) capturing pairwise differences in cerebral activity or model predictions between all pairs of speakers; then, we examined how well the LIN and VLS-derived RDMs correlated with the cerebral RDMs from A1 and the TVAs.…”

“…In recent years, deep neural networks (DNNs) have emerged as a powerful tool for representing complex visual data, such as images (LeCun et al, 2015) or videos (Liu et al, 2020). In the auditory domain, DNNs have been shown to provide valuable representations-so-called feature or latent spaces-for modeling the cerebral processing of sound (brain encoding) (speech: Kell et al, 2018;Millet et al, 2022;Tuckute & Feather, 2023; semantic content: (which was not certified by peer review) is the author/funder. All rights reserved.…”

“…The extent to which the VLS allows linearly predicting the fMRI recordings does not provide insight into the representational geometries, i.e., the differences between the patterns of cerebral activity for speaker identity. We addressed this question by using representational similarity analysis (RSA; Kriegeskorte et al, 2008) (Diedrichsen & Kriegeskorte, 2017;Giordano et al, 2023;Tuckute & Feather, 2023). We built speaker x speaker representational dissimilarity matrices (RDMs) capturing pairwise differences in cerebral activity or model predictions between all pairs of speakers; then, we examined how well the LIN and VLS-derived RDMs correlated with the cerebral RDMs from A1 and the TVAs.…”