Vision is obtained with a continuous motion of the eyes. The kinematic analysis of eye motion, during any visual or ocular task, typically reveals two (kinematic) components: saccades, which quickly replace the visual content in the retinal fovea, and drifts, which slowly scan the image after each saccade. While the saccadic exchange of regions of interest (ROIs) is commonly considered to be included in motor-sensory closed-loops, it is commonly assumed that drifts function in an open-loop manner, that is, independent of the concurrent visual input. Accordingly, visual perception is assumed to be based on a sequence of open-loop processes, each initiated by a saccade-triggered retinal snapshot. Here we directly challenged this assumption by testing the dependency of drift kinematics on concurrent visual inputs using real-time gaze-contingent-display. Our results demonstrate a dependency of the trajectory on the concurrent visual input, convergence of speed to conditionspecific values and maintenance of selected drift-related motor-sensory controlled variables, all strongly indicative of drifts being included in a closed-loop brain-world process, and thus suggesting that vision is inherently a closed-loop process.

Visual perception involves continuously choosing the most prominent inputs while suppressing others. Neuroscientists induce visual competitions in various ways to study why and how the brain makes choices of what to perceive. Recently deep neural networks (DNNs) have been used as models of the ventral stream of the visual system, due to similarities in both accuracy and hierarchy of feature representation. In this study we created non-dynamic visual competitions for humans by briefly presenting mixtures of two images. We then tested feed-forward DNNs with similar mixtures and examined their behavior. We found that both humans and DNNs tend to perceive only one image when presented with a mixture of two. We revealed image parameters which predict this perceptual dominance and compared their predictability for the two visual systems. Our findings can be used to both improve DNNs as models, as well as potentially improve their performance by imitating biological behaviors.

Significance Humans move their eyes continuously to scan their environment. Yet, the role of eye movements in object recognition is not known. In this work, we recorded eye movements of participants attempting to recognize images that are just above and below the threshold of human recognition. To assess the contribution of retinal dynamics, we modeled the activation patterns resulting from the continuous interactions of eye movements with the viewed image. We then trained a classifier to differentiate recognized from unrecognized trials. We show that recognition could be classified only when the continuous interactions between eye movements and the image were used. We suggest that vision is mediated by continuous interactions between eye movements and the environment, resulting in dynamic oculo-retinal coding.

No abstract

24The human visual system perceives its environment via eye movements, whose 25 primary kinematic components are saccades and drifts. Saccades are quick 26 transitions of the gaze from one Region of Interest (ROI) to another and drifts are 27 slower scanning motions in each ROI. While it is accepted that ROI selection depends 28 on the accumulated visual data, drift is commonly considered not to be affected by 29 the acquired visual information. Here we directly tested the latter assumption by 30 testing the dependency of drift kinematics on the concurrent visual inputs. We 31 tracked ocular kinematics in 5 healthy subjects (3 women) while modulating the 32 available visual information via image size and a gaze-contingent display; the latter 33 was used to tunnel vision to a limited window around gaze center. Our results reveal 34 that visual acquisition and the ocular drift movement are linked via a closed-loop 35 dynamic process. This is demonstrated by (i) a dependency of the drift trajectory on 36 the concurrent visual input (ii) condition-specific convergence of the drift speed 37 (within < 100ms) and (iii) maintenance of selected motor-sensory "controlled 38 variables". As these dynamics cannot be accounted for by an open-loop visual scheme, 39 our results suggest that visual acquisition is inherently a closed-loop process. 40 41 3 Author summary 42 Our eyes are nothing like a camera. It has long been known that we are actively 43 scanning our visual environment in order to see. Moreover, it is commonly accepted 44 that our fast eye movements, saccades, are controlled by the brain and are affected 45 by the sensory input. However, our slow eye movements, the ocular drifts, are often 46 ignored when visual acquisition is analyzed. Accordingly, visual processing is 47 typically assumed to be based on computations performed on saccade-triggered 48 snapshots of the retinal state. Our work strongly challenges this model and provides 49 significant evidence for an alternative model, a cybernetic one. We show that the 50 dynamics of the ocular drift do not allow, and cannot be explained by, open loop visual 51 acquisition. Instead, our results suggest that visual acquisition is part of a closed-loop 52 process, which dynamically and continuously links the brain to its environment.53 54

No abstract