When the brain initiates a saccade, it uses a copy of the oculomotor commands to predict the visual consequences: for example, if one fixates a reach target, a peripheral saccade will produce an internal estimate of the new retinal location of the target, a process called remapping. In natural settings, the target likely remains visible after the saccade. So why should the brain predict the sensory consequence of the saccade when after its completion, the image of the target remains visible? We hypothesized that in the post-saccadic period, the brain integrates target position information from two sources: one based on remapping and another based on the peripheral view of the target. The integration of information from these two sources could produce a less variable target estimate than is possible from either source alone. Here, we show that reaching toward targets that were initially foveated and remapped had significantly less variance than reaches relying on peripheral target information. Furthermore, in a more natural setting where both sources of information were available simultaneously, variance of the reaches was further reduced as predicted by integration. This integration occurred in a statistically optimal manner, as demonstrated by the change in integration weights when we manipulated the uncertainty of the post-saccadic target estimate by varying exposure time. Therefore, the brain predicts the sensory consequences of motor commands because it integrates its prediction with the actual sensory information to produce an estimate of sensory space that is better than possible from either source alone.

SUMMARY Inferotemporal cortex (IT) has long been studied as a single pathway dedicated to object vision, but connectivity analysis reveals anatomically distinct channels, through ventral superior temporal sulcus (STSv) and dorsal/ventral inferotemporal gyrus (TEd, TEv). Here, we report a major functional distinction between channels. We studied individual IT neurons in monkeys viewing stereoscopic 3D images projected on a large screen. We used adaptive stimuli to explore neural tuning for 3D abstract shapes ranging in scale and topology from small, closed, bounded objects to large, open, unbounded environments (landscape-like surfaces and cave-like interiors). In STSv, most neurons were more responsive to objects, as expected. In TEd, surprisingly, most neurons were more responsive to 3D environmental shape. Previous studies have localized environmental information to posterior cortical modules. Our results show it is also channeled through anterior IT, where extensive cross-connections between STSv and TEd could integrate object and environmental shape information.

Summary The ventral visual pathway in humans and non-human primates is known to represent object information, including shape and identity [1]. Here, we show the ventral pathway also represents scene structure aligned with the gravitational reference frame in which objects move and interact. We analyzed shape tuning of recently described macaque monkey ventral pathway neurons that prefer scene-like stimuli to objects [2]. Individual neurons did not respond to a single shape class but to a variety of scene elements that are typically aligned with gravity: large planes in the orientation range of ground surfaces under natural viewing conditions, planes in the orientation range of ceilings, extended convex and concave edges in the orientation range of wall/floor/ceiling junctions. For a given neuron, these elements tended to share a common alignment in eye-centered coordinates. Thus, each neuron integrated information about multiple gravity-aligned structures as they would be seen from a specific eye/head orientation. This eclectic coding strategy provides only ambiguous information about individual structures, but explicit information about the environmental reference frame and the orientation of gravity in egocentric coordinates. In the ventral pathway, this could support perceiving and/or predicting physical events involving objects subject to gravity, recognizing object attributes like animacy based on movement not caused by gravity, and/or stabilizing perception of the world against changes in head orientation [3, 4, 5]. Our results, like the recent discovery of object weight representation [6], imply the ventral pathway is involved not just in recognition but also in physical understanding of objects and scenes.