Neural responses to natural and model-matched stimuli reveal distinct computations in primary and nonprimary auditory cortex

Abstract

A central goal of sensory neuroscience is to construct models that can explain neural responses to natural stimuli. As a consequence, sensory models are often tested by comparing neural responses to natural stimuli with model responses to those stimuli. One challenge is that distinct model features are often correlated across natural stimuli, and thus model features can predict neural responses even if they do not in fact drive them. Here, we propose a simple alternative for testing a sensory model: we synthesize a stimulus that yields the same model response as each of a set of natural stimuli, and test whether the natural and “model-matched” stimuli elicit the same neural responses. We used this approach to test whether a common model of auditory cortex—in which spectrogram-like peripheral input is processed by linear spectrotemporal filters—can explain fMRI responses in humans to natural sounds. Prior studies have that shown that this model has good predictive power throughout auditory cortex, but this finding could reflect feature correlations in natural stimuli. We observed that fMRI responses to natural and model-matched stimuli were nearly equivalent in primary auditory cortex (PAC) but that nonprimary regions, including those selective for music or speech, showed highly divergent responses to the two sound sets. This dissociation between primary and nonprimary regions was less clear from model predictions due to the influence of feature correlations across natural stimuli. Our results provide a signature of hierarchical organization in human auditory cortex, and suggest that nonprimary regions compute higher-order stimulus properties that are not well captured by traditional models. Our methodology enables stronger tests of sensory models and could be broadly applied in other domains.

Dynamic time-locking mechanism in the cortical representation of spoken words

“…The crucial question of whether there is something special in the cortical processing of speech has often been approached through careful acoustic matching of linguistic and nonlinguistic stimuli and comparison of their activation patterns in the brain [9][10][11]. The meticulous use of acoustically modified sounds has shown that while primary auditory cortices respond identically to speech and acoustically matched non-speech sounds, responses in nonprimary regions differ [9,12]. However, natural, meaningful and behaviorally relevant sounds might more accurately capture the cortical sensitivity to the most relevant spectrotemporal features for sound processing [13].…”

Metamers of Bayesian computation

“…Since there is no one-to-one correspondence between an optimal decision policy and its underlying BDT elements (likelihood, prior, and cost function), it is possible to design Bayesian metamers that involve different prior-cost pairs but are associated with the same optimal policy. This approach parallels fruitful uses of metamers in perception [37][38][39] and machine learning 52 . The key idea is that an observer whose decisions rely on a model-free optimal policy should 'see' the pairs as metamers because they correspond to the same policy.…”

“…More generally, in decision-making tasks, there are usually numerous pairs of priors and cost functions that, when combined, could lead to indistinguishable decision policies ( Figure 1b). Analogous to the notion of metamers in perception [37][38][39] , we will refer to such pairs of priors and cost functions as prior-cost metamers. Because of the existence of such metamers, it remains an important and unresolved question whether decisions are made based on independently learned priors and cost functions.…”

Spatiotemporal Dynamics of Sound Representations reveal a Hierarchical Progression of Category Selectivity

“…While our stimulus set was designed to minimize confounds between these models (see Methods), we would not necessarily expect a given set of fMRI voxels or MEG sensor patterns to correlate only with one and not the other, both due to the intrinsic correlation between sound acoustics and categories (Theunissen and Elie, 2014) and the fact that neurons throughout primary and nonprimary auditory cortex are responsive to low-and higher-level sound properties (King and Nelken, 2009;Norman-Haignere and McDermott, 2018;Staeren et al, 2009) However, in a hierarchical processing stream, we would expect their respective contributions to vary systematically between regions over time. Thus, we computed the difference between Category and Cochleagram model correlations (Fisher-z normalized to enable direct comparison) with MEG and fMRI patterns.…”

“…We therefore assigned ranks based on independent spatial (functional anatomy) and temporal (fusion-derived peak latency) criteria. For the spatial analysis, we first differentiated primary auditory cortex based on convergent anatomical, histological and functional criteria (Morosan et al, 2001;Norman-Haignere and McDermott, 2018;Sweet et al, 2005), then ranked non-primary areas progressing along and beyond the supratemporal plane. We assigned PAC a rank of 1; TE1.2, PP, and PT (all adjacent to PAC) a rank of 2; TVAx (farther along the posterolateral STG) a rank of 3; LIFG a rank of 4; and FFA, PPA, MPA and LOC a rank of 5.…”