Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions

Abstract

Deep neural networks are commonly used as models of the visual system, but are less explored in audition. Prior work provided examples of audio-trained neural networks that produced good predictions of auditory cortical fMRI responses and exhibited correspondence between model stages and brain regions, but left it unclear whether these results generalize to other neural network models. We evaluated brain-model correspondence for publicly available audio neural network models along with in-house models trained on four different tasks. Most tested models out-predicted previous filter-bank models of auditory cortex, and exhibited systematic model-brain correspondence: middle stages best predicted primary auditory cortex while deep stages best predicted non-primary cortex. However, some state-of-the-art models produced substantially worse brain predictions. The training task influenced the prediction quality for specific cortical tuning properties, with best overall predictions resulting from models trained on multiple tasks. The results suggest the importance of task optimization in constraining brain representations.

Models optimized for real-world tasks reveal the task-dependent necessity of precise temporal coding in hearing

“…This dataset consists of ∼6 million 2-second speech excerpts superimposed on real-world background noise. Training on this dataset has previously been shown to produce models that yield the best current predictions of auditory cortical responses 79 . Models were jointly optimized to classify stimuli according to the word that appeared in the middle of the excerpt (794-way word recognition task) and the talker that produced the utterance (433-way voice recognition task).…”

Exploring an EM-algorithm for banded regression in computational neuroscience

“…This may aid interpretation of encoding analyses [24, 23]. In recent years, encoding models have been used increasingly to model neural responses with features extracted from taskoptimized neural networks [7, 72, 73, 74]. This typically involves very high-dimensional feature sets and hence encoding models where the number of predictors is much larger than the number of observations, i.e., D >> M .…”

Model metamers illuminate divergences between biological and artificial neural networks

“…However, the significance of these visualizations for evaluating neural network models of biological sensory systems has received relatively little attention. One contributing factor may be that model visualizations have often been constrained by added natural image priors or other forms of regularization (77) that help make model visualizations look more natural, but mask the extent to which they otherwise diverge from a perceptually meaningful stimulus. For this reason, we intentionally avoided priors or other regularization when generating model metamers, as they defeat the purpose of the metamer test.…”

“…Thus, most of the variation in metamer recognizability was not captured by standard model-brain comparison benchmarks. We performed an analogous analysis for the auditory models, using a large data set of human auditory cortical responses (74) (fMRI responses to a large set of natural sounds) that had previously been used to evaluate neural network models of the auditory system (5,75). We analyzed voxel responses within four regions of interest in addition to all of auditory cortex, in each case again choosing the best-predicting model stage, measuring the variance it explained in held-out data, and comparing that to the recognizability of the metamers from that stage (Figure 7b).…”