Linking signal detection theory and encoding models to reveal independent neural representations from neuroimaging data

Abstract

Many research questions in visual perception involve determining whether stimulus properties are represented and processed independently. In visual neuroscience, there is great interest in determining whether important object dimensions are represented independently in the brain. For example, theories of face recognition have proposed either completely or partially independent processing of identity and emotional expression. Unfortunately, most previous research has only vaguely defined what is meant by “independence,” which hinders its precise quantification and testing. This article develops a new quantitative framework that links signal detection theory from psychophysics and encoding models from computational neuroscience, focusing on a special form of independence defined in the psychophysics literature: perceptual separability. The new theory allowed us, for the first time, to precisely define separability of neural representations and to theoretically link behavioral and brain measures of separability. The framework formally specifies the relation between these different levels of perceptual and brain representation, providing the tools for a truly integrative research approach. In particular, the theory identifies exactly what valid inferences can be made about independent encoding of stimulus dimensions from the results of multivariate analyses of neuroimaging data and psychophysical studies. In addition, commonly used operational tests of independence are re-interpreted within this new theoretical framework, providing insights on their correct use and interpretation. Finally, we apply this new framework to the study of separability of brain representations of face identity and emotional expression (neutral/sad) in a human fMRI study with male and female participants.

Changes within neural population codes can be inferred from psychophysical threshold studies

“…Despite its successes, inverted encoding modeling makes rather strong implicit assumptions about encoding (e.g., homogeneous population codes, normal neural noise) and about the link between neural activity and neuroimaging measures (a linear measurement model with additive normal noise at each measurement, and independent across measurements) [see 14,8]. In addition, the approach has known identifiability problems.…”

“…For example, face dimensions are often modeled using asymmetric sigmoidal tuning functions [44,5,45,43]. Also, another common assumption for neural channel noise is that it follows a Gaussian distribution [e.g., 6, 7], a common assumption implicit in inverted encoding modeling analyses of neuroimaging data [see 8,14].…”

“…The use of population encoding models has come to dominate the study of human visual neuroscience, serving as a primary tool for making inferences about neural code changes based on indirect measurements, such as psychophysical [e.g., 1,2,3,4,5,6,7,8] and neuroimaging measures [e.g., 9,10,11,12,13,14,15]. One of the primary benefits of these models is that they can be applied to understand the neurocomputational mechanisms of perceptual processes when more invasive methods are not easily available, as is the case in most human neuroscience studies.…”

Improving the validity of neuroimaging decoding tests of invariant and configural neural representation

“…Theoretical Definitions of Invariance at the Neural Level. In our previous work (Soto et al, 2018), we showed that population encoding models provide a formal theoretical framework within which it is possible to define several forms of invariance and independence of neural representations, and link such theoretical definitions to operational tests performed on neuroimaging or psychophysical data. Here we briefly summarize this framework to facilitate understanding of the invariance tests presented later.…”

“…Despite its wide use, recent theoretical work shows that cross-classification and similar operational tests of invariance can lead to invalid conclusions (Soto et al, 2018). In general, it can be shown that no neuroimaging decoding test can provide evidence in favor of invariance.…”