Three-dimensional (3D) representations of the environment are often critical for selecting actions that achieve desired goals. The success of these goal-directed actions relies on 3D sensorimotor transformations that are experience-dependent. Here we investigated the relationships between the robustness of 3D visual representations, choice-related activity, and motor-related activity in parietal cortex. Macaque monkeys performed an eight-alternative 3D orientation discrimination task and a visually guided saccade task while we recorded from the caudal intraparietal area using laminar probes. We found that neurons with more robust 3D visual representations preferentially carried choice-related activity. Following the onset of choice-related activity, the robustness of the 3D representations further increased for those neurons. We additionally found that 3D orientation and saccade direction preferences aligned, particularly for neurons with choice-related activity, reflecting an experience-dependent sensorimotor association. These findings reveal previously unrecognized links between the fidelity of ecologically relevant object representations, choice-related activity, and motor-related activity.

36Reconstructing three-dimensional (3D) scenes from two-dimensional (2D) retinal images is an ill-37 posed problem. Despite this, our 3D perception of the world based on 2D retinal images is 38 seemingly accurate and precise. The integration of distinct visual cues is essential for robust 3D 39 perception in humans, but it is unclear if this mechanism is conserved in non-human primates, 40 and how the underlying neural architecture constrains 3D perception. Here we assess 3D 41 perception in macaque monkeys using a surface orientation discrimination task. We find that 42 perception is generally accurate, but precision depends on the spatial pose of the surface and 43 available cues. The results indicate that robust perception is achieved by dynamically reweighting 44 the integration of stereoscopic and perspective cues according to their pose-dependent 45 reliabilities. They further suggest that 3D perception is influenced by a prior for the 3D orientation 46 statistics of natural scenes. We compare the data to simulations based on the responses of 3D 47 orientation selective neurons. The results are explained by a model in which two independent 48 neuronal populations representing stereoscopic and perspective cues (with perspective signals 49 from the two eyes combined using nonlinear canonical computations) are optimally integrated 50 through linear summation. Perception of combined-cue stimuli is optimal given this architecture. 51However, an alternative architecture in which stereoscopic cues and perspective cues detected 52 by each eye are represented by three independent populations yields two times greater precision 53 than observed. This implies that, due to canonical computations, cue integration for 3D perception 54 is optimized but not maximized. 55 56 Author summary 57Our eyes only sense two-dimensional projections of the world (like a movie on a screen), yet we 58 perceive the world in three dimensions. To create reliable 3D percepts, the human visual system 59 integrates distinct visual signals according to their reliabilities, which depend on conditions such 60 as how far away an object is located and how it is oriented. Here we find that non-human primates 61 similarly integrate different 3D visual signals, and that their perception is influenced by the 3D 62 orientation statistics of natural scenes. Cue integration is thus a conserved mechanism for 63 creating robust 3D percepts by the primate brain. Using simulations of neural population activity, 64 based on neuronal recordings from the same animals, we show that some computations which 65 occur widely in the brain facilitate 3D perception, while others hinder perception. This work 66addresses key questions about how neural systems solve the difficult problem of generating 3D 67 percepts, identifies a plausible neural architecture for implementing robust 3D vision, and reveals 68 how neural computation can simultaneously optimize and curb perception. 69 were linearly summed [10]). However, an alternative architecture in which stereoscopic as we...

Modern neuroscience research often requires the coordination of multiple processes such as stimulus generation, real-time experimental control, as well as behavioral and neural measurements. The technical demands required to simultaneously manage these processes with high temporal fidelity is a barrier that limits the number of labs performing such work. Here we present an open-source, network-based parallel processing framework that lowers this barrier. The Real-Time Experimental Control with Graphical User Interface (REC-GUI) framework offers multiple advantages: (i) a modular design that is agnostic to coding language(s) and operating system(s) to maximize experimental flexibility and minimize researcher effort, (ii) simple interfacing to connect multiple measurement and recording devices, (iii) high temporal fidelity by dividing task demands across CPUs, and (iv) real-time control using a fully customizable and intuitive GUI. We present applications for human, non-human primate, and rodent studies which collectively demonstrate that the REC-GUI framework facilitates technically demanding, behavior-contingent neuroscience research.Editorial note: This article has been through an editorial process in which the authors decide how to respond to the issues raised during peer review. The Reviewing Editor's assessment is that all the issues have been addressed (see decision letter).

The visual system must reconstruct the dynamic, three-dimensional (3D) world from ambiguous two-dimensional (2D) retinal images. In this review, we synthesize current literature on how the visual system of nonhuman primates performs this transformation through multiple channels within the classically defined dorsal (where) and ventral (what) pathways. Each of these channels is specialized for processing different 3D features (e.g., the shape, orientation, or motion of objects, or the larger scene structure). Despite the common goal of 3D reconstruction, neurocomputational differences between the channels impose distinct information-limiting constraints on perception. Convergent evidence further points to the little-studied area V3A as a potential branchpoint from which multiple 3D-fugal processing channels diverge. We speculate that the expansion of V3A in humans may have supported the emergence of advanced 3D spatial reasoning skills. Lastly, we discuss future directions for exploring 3D information transmission across brain areas and experimental approaches that can further advance the understanding of 3D vision. Expected final online publication date for the Annual Review of Vision Science, Volume 9 is September 2023. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Visually guided behaviors require the brain to transform ambiguous retinal images into object-level spatial representations and implement sensorimotor transformations. These processes are supported by the dorsal 'where' pathway. However, the specific functional contributions of areas along this pathway remain elusive due in part to methodological differences across studies. We previously showed that macaque caudal intraparietal (CIP) area neurons possess robust three-dimensional (3D) visual representations, carry choice- and saccade-related activity, and exhibit experience-dependent sensorimotor associations (Chang et al., 2020b). Here, we used a common experimental design to reveal parallel processing, hierarchical transformations, and the formation of sensorimotor associations along the 'where' pathway by extending the investigation to V3A, a major feedforward input to CIP. Higher-level 3D representations and choice-related activity were more prevalent in CIP than V3A. Both areas contained saccade-related activity that predicted the direction/timing of eye movements. Intriguingly, the time-course of saccade-related activity in CIP aligned with the temporally integrated V3A output. Sensorimotor associations between 3D orientation and saccade direction preferences were stronger in CIP than V3A, and moderated by choice signals in both areas. Together, the results explicate parallel representations, hierarchical transformations, and functional associations of visual and saccade-related signals at a key juncture in the 'where' pathway.

Reconstructing three-dimensional (3D) scenes from two-dimensional (2D) retinal images is an ill-posed problem. Despite this, 3D perception of the world based on 2D retinal images is seemingly accurate and precise. The integration of distinct visual cues is essential for robust 3D perception in humans, but it is unclear whether this is true for non-human primates (NHPs). Here, we assessed 3D perception in macaque monkeys using a planar surface orientation discrimination task. Perception was accurate across a wide range of spatial poses (orientations and distances), but precision was highly dependent on the plane's pose. The monkeys achieved robust 3D perception by dynamically reweighting the integration of stereoscopic and perspective cues according to their pose-dependent reliabilities. Errors in performance could be explained by a prior resembling the 3D orientation statistics of natural scenes. We used neural network simulations based on 3D orientation-selective neurons recorded from the same monkeys to assess how neural computation might constrain perception. The perceptual data were consistent with a model in which the responses of two independent neuronal populations representing stereoscopic cues and perspective cues (with perspective signals from the two eyes combined using nonlinear canonical computations) were optimally integrated through linear summation. Perception of combined-cue stimuli was optimal given this architecture. However, an alternative architecture in which stereoscopic cues, left eye perspective cues, and right eye perspective cues were represented by three independent populations yielded two times greater precision than the monkeys. This result suggests that, due to canonical computations, cue integration for 3D perception is optimized but not maximized. Significance StatementOur eyes sense two-dimensional (2D) projections of the world, like a movie on a screen, but we perceive the world as three-dimensional (3D). Here, we show that non-human primates (NHPs), like humans, achieve more precise 3D vision by perceptually integrating distinct 3D cues. We also present evidence that perception is influenced by 3D natural scene statistics, and that priors over 3D orientation are subjectively encoded. Using simulations, we examine how neural computation can constrain 3D perception and estimate that perception is half as precise as theoretically possible. Our findings suggest that the concurrence of multiple canonical computations simultaneously optimizes and curbs 3D visual perception, and highlight that what constitutes optimal task performance depends on the underlying neural architecture.

35Modern neuroscience research often requires the coordination of multiple processes such as 36 stimulus generation, real-time experimental control, as well as behavioral and neural 37 measurements. The technical demands required to simultaneously manage these processes with 38 high temporal fidelity limits the number of labs capable of performing such work. Here we present 39 an open-source network-based parallel processing framework that eliminates these barriers. The 40Real-Time Experimental Control with Graphical User Interface (REC-GUI) framework offers 41 multiple advantages: (i) a modular design agnostic to coding language(s) and operating system(s) 42 that maximizes experimental flexibility and minimizes researcher effort, (ii) simple interfacing to 43 connect measurement and recording devices, (iii) high temporal fidelity by dividing task demands 44 across CPUs, and (iv) real-time control using a fully customizable and intuitive GUI. Testing 45 results demonstrate that the REC-GUI framework facilitates technically demanding, behavior-46 contingent neuroscience research. Sample code and hardware configurations are downloadable, 47

Robust control of action relies on the ability to perceive, integrate, and act on information from multiple sensory modalities including vision and proprioception. How does the brain combine sensory information to regulate ongoing mechanical interactions between the body and its physical environment? Some behavioral studies suggest that the rules governing multisensory integration for action may differ from the maximum likelihood estimation rules that appear to govern multisensory integration for many perceptual tasks. We used functional magnetic resonance (MR) imaging techniques, a MR-compatible robot, and a multisensory feedback control task to test that hypothesis by investigating how neural mechanisms involved in regulating hand position against mechanical perturbation respond to the presence and fidelity of visual and proprioceptive information. Healthy human subjects rested supine in a MR scanner and stabilized their wrist against constant or pseudo-random torque perturbations imposed by the robot. These two stabilization tasks were performed under three visual feedback conditions: “No-vision”: Subjects had to rely solely on proprioceptive feedback; “true-vision”: visual cursor and hand motions were congruent; and “random-vision”: cursor and hand motions were uncorrelated in time. Behaviorally, performance errors accumulated more quickly during trials wherein visual feedback was absent or incongruous. We analyzed blood-oxygenation level-dependent (BOLD) signal fluctuations to compare task-related activations in a cerebello-thalamo-cortical neural circuit previously linked with feedback stabilization of the hand. Activation in this network varied systematically depending on the presence and fidelity of visual feedback of task performance. Addition of task related visual information caused activations in the cerebello-thalamo-cortical network to expand into neighboring brain regions. Specific loci and intensity of expanded activity depended on the fidelity of visual feedback. Remarkably, BOLD signal fluctuations within these regions correlated strongly with the time series of proprioceptive errors—but not visual errors—when the fidelity of visual feedback was poor, even though visual and hand motions had similar variability characteristics. These results provide insight into the neural control of the body’s physical interactions with its environment, rejecting the standard Gaussian cue combination model of multisensory integration in favor of models that account for causal structure in the sensory feedback.