Visual function depends on the accuracy of signals carried by visual cortical neurons. Combining information across neurons should improve this accuracy because single neuron activity is variable. We examined the reliability of information inferred from populations of simultaneously recorded neurons in macaque primary visual cortex. We considered a decoding framework that computes the likelihood of visual stimuli from a pattern of population activity by linearly combining neuronal responses, and tested this framework for orientation estimation and discrimination. We derived a simple parametric decoder assuming neuronal independence, and a more sophisticated empirical decoder that learned the structure of the measured neuronal response distributions, including their correlated variability. The empirical decoder used the structure of these response distributions to perform better than its parametric variant, showing that their structure contains critical information for sensory decoding. Our work shows how neuronal responses can best be used to inform perceptual decision-making.A central question in computational and systems neuroscience is how signals carried by sensory neurons support perceptual judgments. To put these signals to use in behavioral tasks, the brain must accurately decode the responses of neurons encoding a sensory signal [1][2][3][4][5][6] . There are many reasons to think that the brain performs this decoding by combining signals from populations of neurons. First, the variability of a neuron's response to repeated presentations of the same stimulus is considerable, and limits what can be inferred from individual neurons [7][8][9][10] . Second, the synaptic architecture of visual cortex 11 makes it virtually impossible for a signal from any one neuron to lead directly to a behavioral outcome. Third, perceptual judgments are only weakly correlated with variations in the response of single neurons in sensory cortex 12,13 .Many studies have focused on how much information about a stimulus is encoded in population activity 2,[14][15][16][17][18] . Specifying how that information could be extracted from the population code is a challenge 19,20 . This latter issue is the problem of population decoding, a computational question that investigates how, and with what accuracy, a sensory stimulus can be inferred from the responses of neuronal populations, for example by a downstream neuron. Previous decoding studies have examined how a single stimulus estimate could be directly inferred from population responses, for instance by calculating the population vector 21 or the least-squares error estimator 22 . Sensory decoding is, however, more general than reading out sensory responses for a particular psychophysical task such as estimation. A decoding framework should provide a rigorous account of the reliability of information embedded in population responses for a wide range of psychophysical tasks, including estimation and discrimination. We formulate the problem of sensory decoding as inferencec...

No abstract