Modeling peripheral visual acuity enables discovery of gaze strategies at multiple time scales during natural scene search

Abstract

Like humans, monkeys make saccades nearly three times a second. To understand the factors guiding this frequent decision, computational models of vision attempt to predict fixation locations using bottom-up visual features and top-down goals. How do the relative influences of these factors evolve over multiple time scales? Here we analyzed visual features at fixations using a retinal transform that provides realistic visual acuity by suitably degrading visual information in the periphery. In a task in which monkeys searched for a Gabor target in natural scenes, we characterized the relative importance of bottom-up and task-relevant influences by decoding fixated from nonfixated image patches based on visual features. At fast time scales, we found that search strategies can vary over the course of a single trial, with locations of higher saliency, target-similarity, edge–energy, and orientedness looked at later on in the trial. At slow time scales, we found that search strategies can be refined over several weeks of practice, and the influence of target orientation was significant only in the latter of two search tasks. Critically, these results were not observed without applying the retinal transform. Our results suggest that saccade-guidance strategies become apparent only when models take into account degraded visual representation in the periphery.

Feature-based attention and spatial selection in frontal eye fields during natural scene search

“…In practice, to avoid sensitivity to the precise phase of the Gabor, we convolved the target as well as its 90°phase-shifted version with the scene. We then took the sum of squares of the convolutions (Ramkumar et al 2015; see Fig. 2A).…”

“…Edge-energy map. Edge energy of natural scenes is known to influence the fixation choice of both humans and monkeys in visual search (Ramkumar et al 2015;Rajashekar et al 2003;Ganguli et al 2010). Therefore, we also computed edge energy as a potential feature that may influence fixation choice, defined as the sum of squares of the vertical and horizontal edge gradients ( Fig.…”

“…Therefore, we also computed edge energy as a potential feature that may influence fixation choice, defined as the sum of squares of the vertical and horizontal edge gradients ( Fig. 2B; for details, see Ramkumar et al 2015). For the fly search task, we calculated the energy maps for each of the RGB color channels and summed them.…”

“…B: edge-energy maps were computed by taking the sum of squares of horizontal and vertical edge gradients. Before computing the relevance and energy maps, the image was degraded in accordance with decreasing visual acuity in the periphery, with respect to an example fixation location shown as a yellow crosshair [adapted from Ramkumar et al 2015]. of saccade endpoints (fixations), in conjunction with the model-based feature maps (described above) for each image viewed by the monkeys. If feature maps indeed predict fixation locations, saccade endpoints should fall on image patches with higher values of the respective maps than those not drawing saccades.…”

“…Although it is well known that visual acuity is strongest at the fovea and falls off with eccentricity, predictive models of gaze behavior have not taken it into account. Indeed, we have recently shown that modeling visual acuity has enabled the discovery of gaze strategies at different time scales during visual search (Ramkumar et al 2015). Therefore, to model decreasing visual acuity with peripheral distance, we processed the stimuli before computing the feature maps.…”