Humans excel at inferring information about 3D scenes from their 2D images projected on the retinas, using a wide range of depth cues. One example of such inference is the tendency for observers to perceive lighter image regions as closer. This psychophysical behavior could have an ecological basis because nearer regions tend to be lighter in natural 3D scenes. Here, we show that an analogous association exists between the relative luminance and binocular disparity preferences of neurons in macaque primary visual cortex. The joint coding of relative luminance and binocular disparity at the neuronal population level may be an integral part of the neural mechanisms for perceptual inference of depth from images.vision | neuronal coding | stereopsis | illusion T he properties of the natural environment are essential to understanding behavior (1). The study of the statistics of natural images has been instrumental in advancing our understanding of the visual system. Examples are numerous: Center-surround receptive fields of retinal ganglion cells can be understood in terms of their whitening effects on highly self-correlated natural image signals (2). In primary visual cortex (V1), the need to further separate visual signals into their underlying causes provides a potential sparse code explanation for the wavelet-like receptive fields of V1 simple cells (3, 4).Thus far, knowledge of the statistics of natural images has been most useful for understanding how images are represented and transmitted in the early visual pathway (5). However, efficient image representation is only one of many goals of the visual system. To effectively study perceptual inference in the visual system, we must have an understanding of the joint statistics of images together with their perceptual goals. In particular, to understand how depth and 3D shape are inferred in the brain, we need to first understand the statistical trends that exist between natural images and their underlying 3D structure.Statistical trends in natural scenes often manifest as a perceptual bias in psychophysical studies (6-12). One perceptual bias that has been observed is that, all other things being equal, humans perceive lighter image regions as being closer. This perceptual effect was first discovered by Leonardo da Vinci who said, "Among bodies equal in size and distance, that which shines the more brightly seems to the eye nearer," (ref. 13, p. 332). Over the last century, a number of psychophysical studies have characterized this relationship between relative luminance and depth (14-21). We have previously shown that a corresponding negative correlation (r = −0.14) between image intensity and depth (r = −0.24 for log intensity vs. log depth) was found in the statistics of natural scenes (22). This result suggests that the perceptual bias in the psychophysical tests has an ecological basis and is revealing a statistical trend stored within our visual system. Such a trend could be used by the visual system to infer depth in a scene when binocular and other visual ...