A neural representation of depth from motion parallax in macaque visual cortex

Abstract

Perception of depth is a fundamental challenge for the visual system, particularly for observers moving through their environment. The brain makes use of multiple visual cues to reconstruct the three-dimensional structure of a scene. One potent cue, motion parallax, frequently arises during translation of the observer because the images of objects at different distances move across the retina with different velocities. Human psychophysical studies have demonstrated that motion parallax can be a powerful depth cue 1-5 , and motion parallax appears to be heavily exploited by animal species that lack highly developed binocular vision 6-8 . However, little is known about the neural mechanisms that underlie this capacity. We used a virtual-reality system to translate macaque monkeys while they viewed motion parallax displays that simulated objects at different depths. We show that many neurons in the middle temporal (MT) area signal the sign of depth (i.e., near vs. far) from motion parallax in the absence of other depth cues. To achieve this, neurons must combine visual motion with extra-retinal (non-visual) signals related to the animal's movement. Our findings suggest a new neural substrate for depth perception, and demonstrate a robust interaction of visual and non-visual cues in area MT. Combined with previous studies that implicate area MT in depth perception based on binocular disparities 9-12 , our results suggest that MT contains a more general representation of three dimensional space that leverages multiple cues.Humans can make precise judgments of depth based on motion parallax, the relative retinal image motion between objects at different distances 1-5 . However, motion parallax alone is not sufficient to specify the sign of depth, that is whether an object is near or far relative to the plane of fixation 13-15 . Rather, the direction of image motion relative to observer motion is crucial to specifying depth-sign (Fig. 1). Objects located nearer than the plane of fixation sweep across the visual field in a direction opposite to head translation (Fig. 1, black arrows). In contrast, objects located farther than the plane of fixation move in the same direction as the head (Fig 1, white arrows). Thus, image motion in a particular direction (e.g. leftward) can be associated with either a near (Fig. 1a) or far (Fig. 1b) object. Under some conditions, the brain could interpret such ambiguous visual motion by using other cues such as occlusion, size, or perspective. However, to compute depth-sign in the absence of these pictorial cues, the visual system needs access to extra-retinal signals related to observer movement. We exploited this fact to probe for a neural correlate of depth from motion parallax.We performed extracellular microelectrode recordings in area MT of two macaque monkeys that were trained to maintain visual fixation on a world-fixed target while being translated by

The neural basis of depth perception from motion parallax

“…With this stimulus paradigm, Nadler et al [35] recorded extracellularly from neurons in the middle temporal (MT) area of macaque monkeys. Area MT is well known for its roles in visual motion perception [36], and also contains many neurons that are selective for depth defined by binocular disparity [37][38][39].…”

“…This led Nadler et al [35] to design visual stimuli (figure 2c) that are depth-sign ambiguous when viewed by a stationary observer, but which yield clear percepts of depth-sign (near versus far) when viewed by a translating macaque that is trained to fixate a world-fixed target (viewing geometry illustrated in figure 2a,b). In this design, extra-retinal signals related to head or eye movements are necessary to disambiguate the depth-sign of the visual stimuli, thus providing a much more effective method to probe for a neural correlate of perceived depth.…”

“…figure 3b, black). Because the retinal image motion over the MT receptive field is the same in these two stimulus conditions (assuming accurate pursuit tracking, [35]), the difference in response between the retinal motion and motion parallax conditions must reflect the action of extra-retinal signals related to head translation or eye rotation. Note that the data shown by the green and orange curves in figure 3 will be described in a subsequent section.…”

“…Thin curves represent average eye position and velocity traces for a single session, in equivalent stimulus units. (Reproduced from Nadler et al [35].) related to head or eye movements.…”

“…In other words, responses of the MT neuron are depth-sign ambiguous in the absence of extra-retinal signals Figure 2. Schematic of motion parallax and stimulus design used by Nadler et al [35] to establish neural correlates of depth from motion parallax. (a) As the head moves to the right, the image of a near object moves leftward, while the image of a far object moves rightward.…”