Finding any Waldo with zero-shot invariant and efficient visual search

Abstract

Searching for a target object in a cluttered scene constitutes a fundamental challenge in daily vision. Visual search must be selective enough to discriminate the target from distractors, invariant to changes in the appearance of the target, efficient to avoid exhaustive exploration of the image, and must generalize to locate novel target objects with zero-shot training. Previous work on visual search has focused on searching for perfect matches of a target after extensive category-specific training. Here, we show for the first time that humans can efficiently and invariantly search for natural objects in complex scenes. To gain insight into the mechanisms that guide visual search, we propose a biologically inspired computational model that can locate targets without exhaustive sampling and which can generalize to novel objects. The model provides an approximation to the mechanisms integrating bottom-up and top-down signals during search in natural scenes.

Beyond the feedforward sweep: feedback computations in the visual cortex

“…Related recent work by Adeli and Zelinsky provided a biologically inspired implementation of biased competition theory, whereby the multiple objects in a display compete with each other for attention and a top-down signal is used to disambiguate and bias this competition in favor of the sought target. 125 Such feature-based modulation is more efficient when applied at later stages of the visual hierarchy, 124,126 which is consistent with physiological observations showing that both spatial and feature-based attention is considerably weaker in early visual cortical areas compared with higher visual cortical areas.…”

“…In stark contrast, Zhang et al . show that their model can rapidly find target objects after a single exposure to them 124 …”

“…Several computational models have been proposed recently to capture how top‐down signals modulate processing of an image and guide eye movements during visual search. Inspired by the neurophysiology of visual search, Zhang and colleagues built a simple architecture consisting of a DCNN, which aims to mimic the extraction of features along the VVC, and a PFC‐like module that stores information about the sought target and provides top‐down feature‐based attentional modulation onto the VC 124 . Combining the bottom‐up features with top‐down target modulation led to the creation of an attention map that dictates the location of the next saccade in a winner‐take‐all fashion.…”

“…Inspired by the neurophysiology of visual search, Zhang and colleagues built a simple architecture consisting of a DCNN, which aims to mimic the extraction of features along the VVC, and a PFC-like module that stores information about the sought target and provides top-down feature-based attentional modulation onto the VC. 124 Combining the bottom-up features with top-down target modulation led to the creation of an attention map that dictates the location of the next saccade in a winner-take-all fashion. The model was able to provide a reasonable approximation to both the number and spatiotemporal sequence of eye movements that humans executed during visual search tasks spanning a wide range of difficulty levels.…”

Eye movement characteristics in schizophrenia: A recent update with clinical implications

“…Some eye movement characteristics, such as exploratory eye movements, are known to change with development and can be changed with reinforcement learning . Perhaps by combining findings from recent mathematical models of visual search and exploration, development of “eye movement training programs” for schizophrenia aimed at improving visual cognition or social functioning may become possible in the future. The development of biomarkers that can be used for clinical and personal recovery would be of great benefit for both the individual with mental illness and their supporters.…”