Salience Models: A Computational Cognitive Neuroscience Review

Abstract

The seminal model by Laurent Itti and Cristoph Koch demonstrated that we can compute the entire flow of visual processing from input to resulting fixations. Despite many replications and follow-ups, few have matched the impact of the original model-so what made this model so groundbreaking? We have selected five key contributions that distinguish the original salience model by Itti and Koch; namely, its contribution to our theoretical, neural, and computational understanding of visual processing, as well as the spatial and temporal predictions for fixation distributions. During the last 20 years, advances in the field have brought up various techniques and approaches to salience modelling, many of which tried to improve or add to the initial Itti and Koch model. One of the most recent trends has been to adopt the computational power of deep learning neural networks; however, this has also shifted their primary focus to spatial classification. We present a review of recent approaches to modelling salience, starting from direct variations of the Itti and Koch salience model to sophisticated deep-learning architectures, and discuss the models from the point of view of their contribution to computational cognitive neuroscience.Vision 2019, 3, 56 2 of 24 a true sign of its importance in defining this field. We provide a review of the different approaches in modelling visual attention based on their contribution to computational cognitive neuroscience. We limit our review to models that were directly influenced by the Itti and Koch algorithm, and cover examples that model object attention, top-down influence, information theory, dual stream models, and conclude with recent advances in deep learning salience classifiers. We also include other methods of achieving the main goal: modelling image salience and the way it results in shifts of attention. The main questions this review tries to address are what contribution did these models make in our goal to explain human visual attention using computer simulations, and what directions are available for the next generation of models?In this review, we approach the salience problem from a computational cognitive neuroscience perspective, which implies a combination of theoretical and methodological concepts from several domains. Such a perspective suggests that the perfect salience model should be based on a strong theoretical foundation, model the neurobiological processes underlying visual saliency, use explicit computational tools as a means of modelling these processes, and be generative by taking both spatial and temporal predictions of visual salience into account.Studies of human visual salience have led to the creation of hundreds of computationally valid models, however, most of these models do not focus on all of the abovementioned components of salience simultaneously. We understand that with so many parameters to account for, like combining a broad cognitive theoretical scope and a focused precise neural approach, it is almost impossible to avoid trade-offs. Never...

Temporal Limitations of the Standard Leaky Integrate and Fire Model

“…The distribution produced by the resulting parameter space was not close to the ground truth of human data (D = 0.19327, p < 2.2 × 10 −16 ; see Figure 9). (16)(17)(18)(19) data points for each image). On the right: all saccades (542-794 data points for each image, mean number 661).…”

“…The distributions of saccades' latency for each image in the dataset. On the left: only initial saccades(16)(17)(18)(19) data points for each image). On the right: all saccades (542-794 data points for each image, mean number 661).…”

EyeTrackUAV2: A Large-Scale Binocular Eye-Tracking Dataset for UAV Videos

“…This is known as top-down or endogenous attention. A strict division is slightly inaccurate in that both top-down and bottom-up processes are triggered during a visual stimuli in a very intricate interaction [17,18].…”

“…Salience analyses rest on the relation of visual attention to eye movements, and these latter are obtained through gaze collection with eye-trackers [20]. Saliency predictions help to understand computational cognitive neuroscience as it reveals attentional behaviors and systematic viewing tendencies such as center bias [17]. Multiple applications derive from saliency predictions such as compression [21], content-aware re-targeting, object segmentation [22], and detection [23,24].…”

What Neuroscientific Studies Tell Us about Inhibition of Return

“…In real-world applications of visual search, such as robot navigation, inhibitory algorithms must be implemented in order to avoid perseverance on highly salient stimuli. However, such computations are normally implemented by simply reducing the salience of previously attended stimuli to zero for a few seconds [70], which is clearly not how the primate brain accomplishes the task (see [71] for a recent review of such salience models).…”