SummaryPrefrontal cortex is thought to play a fundamental role in flexible, context-dependent behavior, but the exact nature of the computations underlying this role remains largely mysterious. In particular, individual prefrontal neurons often generate remarkably complex responses that defy deep understanding of their contribution to behavior. Here we study prefrontal cortex in monkeys trained to flexibly select and integrate noisy sensory inputs towards a choice. We find that the observed complexity and functional roles of single neurons are readily understood in the framework of a dynamical process unfolding at the level of the population. The population dynamics can be reproduced by a trained recurrent neural network, which suggests a previously unknown mechanism for selection and integration of task-relevant inputs. This mechanism implies that selection and integration are two aspects of a single dynamical process unfolding within the same prefrontal circuits, and potentially provides a novel, general framework for understanding context-dependent computations.

Neurons in area MT (V5) are selective for the direction of visual motion. In addition, many are selective for the motion of complex patterns independent of the orientation of their components, a behavior not seen in earlier visual areas. We show that the responses of MT cells can be captured by a linear-nonlinear model that operates not on the visual stimulus, but on the afferent responses of a population of nonlinear V1 cells. We fit this cascade model to responses of individual MT neurons and show that it robustly predicts the separately measured responses to gratings and plaids. The model captures the full range of pattern motion selectivity found in MT. Cells that signal pattern motion are distinguished by having convergent excitatory input from V1 cells with a wide range of preferred directions, strong motion opponent suppression and a tuned normalization that may reflect suppressive input from the surround of V1 cells.

We can claim that we know what the visual system does once we can predict neural responses to arbitrary stimuli, including those seen in nature. In the early visual system, models based on one or more linear receptive fields hold promise to achieve this goal as long as the models include nonlinear mechanisms that control responsiveness, based on stimulus context and history, and take into account the nonlinearity of spike generation. These linear and nonlinear mechanisms might be the only essential determinants of the response, or alternatively, there may be additional fundamental determinants yet to be identified. Research is progressing with the goals of defining a single "standard model" for each stage of the visual pathway and testing the predictive power of these models on the responses to movies of natural scenes. These predictive models represent, at a given stage of the visual pathway, a compact description of visual computation. They would be an invaluable guide for understanding the underlying biophysical and anatomical mechanisms and relating neural responses to visual perception.Key words: contrast; lateral geniculate nucleus; luminance; primary visual cortex; receptive field; retina; visual system; natural imagesThe ultimate test of our knowledge of the visual system is prediction: we can say that we know what the visual system does when we can predict its response to arbitrary stimuli. How far are we from this end result? Do we have a "standard model" that can predict the responses of at least some early part of the visual system, such as the retina, the lateral geniculate nucleus (LGN), or primary visual cortex (V1)? Does such a model predict responses to stimuli encountered in the real world?A standard model existed in the early decades of visual neuroscience, until the 1990s: it was given by the linear receptive field. The linear receptive field specifies a set of weights to apply to images to yield a predicted response. A weighted sum is a linear operation, so it is simple and intuitive. Moreover, linearity made the receptive field mathematically tractable, allowing the fruitful marriage of visual neuroscience with image processing (Robson, 1975) and with linear systems analysis (De Valois and De Valois, 1988). It also provided a promising parallel with research in visual perception (Graham, 1989). Because it served as a standard model, the receptive field could be used to decide which findings were surprising and which were not: if a phenomenon was not predictable from the linear receptive field, it was particularly worthy of publication.Research aimed at testing the linear receptive field led to the discovery of important nonlinear phenomena, which cannot be explained by a linear receptive field alone. These phenomena have been discovered at all stages of the early visual system, including the retina (for review, see Shapley and Enroth-Cugell, 1984;Demb, 2002), the LGN (for review, see Carandini, 2004), and area V1 (for review, see Carandini et al., 1999;Fitzpatrick, 2000;Albright and Stoner,...

The early visual system is endowed with adaptive mechanisms that rapidly adjust gain and integration time based on the local luminance (mean intensity) and contrast (standard deviation of intensity relative to the mean). Here we show that these mechanisms are matched to the statistics of the environment. First, we measured the joint distribution of luminance and contrast in patches selected from natural images and found that luminance and contrast were statistically independent of each other. This independence did not hold for artificial images with matched spectral characteristics. Second, we characterized the effects of the adaptive mechanisms in lateral geniculate nucleus (LGN), the direct recipient of retinal outputs. We found that luminance gain control had the same effect at all contrasts and that contrast gain control had the same effect at all mean luminances. Thus, the adaptive mechanisms for luminance and contrast operate independently, reflecting the very independence encountered in natural images.

The responses of neurons in lateral geniculate nucleus (LGN) exhibit powerful suppressive phenomena such as contrast saturation, size tuning, and masking. These phenomena cannot be explained by the classical center-surround receptive field and have been ascribed to a variety of mechanisms, including feedback from cortex. We asked whether these phenomena might all be explained by a single mechanism, contrast gain control, which is inherited from retina and possibly strengthened in thalamus. We formalized an intuitive model of retinal contrast gain control that explicitly predicts gain as a function of local contrast. In the model, the output of the receptive field is divided by the output of a suppressive field, which computes the local root-mean-square contrast. The model provides good fits to LGN responses to a variety of stimuli; with a single set of parameters, it captures saturation, size tuning, and masking. It also correctly predicts that responses to small stimuli grow proportionally with contrast: were it not for the suppressive field, LGN responses would be linear. We characterized the suppressive field and found that it is similar in size to the surround of the classical receptive field (which is eight times larger than commonly estimated), it is not selective for stimulus orientation, and it responds to a wide range of frequencies, including very low spatial frequencies and high temporal frequencies. The latter property is hardly consistent with feedback from cortex. These measurements thoroughly describe the visual properties of contrast gain control in LGN and provide a parsimonious explanation for disparate suppressive phenomena.

Recent work has suggested that prefrontal cortex (PFC) plays a key role in context-dependent perceptual decision-making. Here we address that role using a new method for identifying taskrelevant dimensions of neural population activity. Specifically, we show that PFC has a multidimensional code for context, decisions, and both relevant and irrelevant sensory information. Moreover, these representations evolve in time, with an early linear accumulation phase followed by a phase with rotational dynamics. We identify the dimensions of neural activity associated with these phases, and show that they do not arise from distinct populations, but of a single population with broad tuning characteristics. Finally, we use model-based decoding to show that the transition from linear to rotational dynamics coincides with a plateau in decoding accuracy, revealing that rotational dynamics in PFC preserve sensory choice information for the duration of the stimulus integration period.

Functional models of the early visual system should predict responses not only to simple artificial stimuli but also to sequences of complex natural scenes. An ideal testbed for such models is the lateral geniculate nucleus (LGN). Mechanisms shaping LGN responses include the linear receptive field and two fast adaptation processes, sensitive to luminance and contrast. We propose a compact functional model for these mechanisms that operates on sequences of arbitrary images. With the same parameters that fit the firing rate responses to simple stimuli, it predicts the bulk of the firing rate responses to complex stimuli, including natural scenes. Further improvements could result by adding a spiking mechanism, possibly one capable of bursts, but not by adding mechanisms of slow adaptation. We conclude that up to the LGN the responses to natural scenes can be largely explained through insights gained with simple artificial stimuli.

In the early visual system, a contrast gain control mechanism sets the gain of responses based on the locally prevalent contrast. The measure of contrast used by this adaptation mechanism is commonly assumed to be the standard deviation of light intensities relative to the mean (root-mean-square contrast). A number of alternatives, however, are possible. For example, the measure of contrast might depend on the absolute deviations relative to the mean, or on the prevalence of the darkest or lightest intensities. We investigated the statistical computation underlying this measure of contrast in the cat's lateral geniculate nucleus, which relays signals from retina to cortex. Borrowing a method from psychophysics, we recorded responses to white noise stimuli whose distribution of intensities was precisely varied. We varied the standard deviation, skewness, and kurtosis of the distribution of intensities while keeping the mean luminance constant. We found that gain strongly depends on the standard deviation of the distribution. At constant standard deviation, moreover, gain is invariant to changes in skewness or kurtosis. These findings held for both ON and OFF cells, indicating that the measure of contrast is independent of the range of stimulus intensities signaled by the cells. These results confirm the long-held assumption that contrast gain control computes root-mean-square contrast. They also show that contrast gain control senses the full distribution of intensities and leaves unvaried the relative responses of the different cell types. The advantages to visual processing of this remarkably specific computation are not entirely known.