Where Should Saliency Models Look Next?

Abstract

Abstract. Recently, large breakthroughs have been observed in saliency modeling. The top scores on saliency benchmarks have become dominated by neural network models of saliency, and some evaluation scores have begun to saturate. Large jumps in performance relative to previous models can be found across datasets, image types, and evaluation metrics. Have saliency models begun to converge on human performance? In this paper, we re-examine the current state-of-the-art using a finegrained analysis on image types, individual images, and image regions. Using experiments to gather annotations for high-density regions of human eye fixations on images in two established saliency datasets, MIT300 and CAT2000, we quantify up to 60% of the remaining errors of saliency models. We argue that to continue to approach human-level performance, saliency models will need to discover higher-level concepts in images: text, objects of gaze and action, locations of motion, and expected locations of people in images. Moreover, they will need to reason about the relative importance of image regions, such as focusing on the most important person in the room or the most informative sign on the road. More accurately tracking performance will require finer-grained evaluations and metrics. Pushing performance further will require higher-level image understanding.

Preferential Processing of Social Features and Their Interplay with Physical Saliency in Complex Naturalistic Scenes

“…Please note that even though GBVS was implemented in the current study, all of our major results were qualitatively similar both with the traditional Itti and Koch model (Itti et al, 1998; Itti and Koch, 2000) and with a very recent model (Boolean Map Saliency; Zhang and Sclaroff, 2016) which achieved high performance on the MIT Saliency Benchmark website (Bylinskii et al, 2016). …”

“…The GBVS model was implemented in the current study because it was previously demonstrated to be one of the best performing saliency-based attention models which is biologically plausible, available with Matlab ® source code, and applicable without initial machine learning of feature weights by means of ground truth training data (Harel et al, 2007; Judd et al, 2012; Borji and Itti, 2013; see also the MIT Saliency Benchmark website of Bylinskii et al, 2016) 2 . In GBVS, those locations of a visual input that are most different from their surrounding in terms of their low-level features (i.e., color, intensity, and orientation) are calculated on various spatial scales using graph-based dissimilarity representations which are interpreted as Markov chains.…”

Fabric defect detection based on saliency histogram features

“…There are hundreds of saliency computational models, and different models produce saliency maps with different quality. Among the various model approaches, the deep learning–based models can produce saliency maps with the best quality . However, the deep learning–based models require massive amounts of data, and it is difficult to collect a huge database of patterned texture to train the deep network.…”

Predicting eye movements from deep neural network activity decoded from fMRI responses to natural scenes

“…To mitigate this problem, selective spatial attention allows the visual system to modulate perceptual and cognitive processing towards important regions of space 1,2 . Computational modeling of selective spatial attention has been an area of focused research for the past 20 years, and numerous computational hypotheses have been proposed regarding the features, representations, and operations necessary to guide spatial attention (for reviews, see [3][4][5] ). Central to most selective spatial attention models is a spatial priority map, a representation that tags the priority of different regions in space for the allocation of covert attention and overt attention such as eye movements [6][7][8] .…”

“…Recently, selective spatial attention models using deep convolutional neural network (CNN) architectures have yielded state-of-the-art prediction of human eye movement patterns (for a review see 5 ). Inspired by the importance of object information in guiding spatial attention, these models extract attention-relevant features from images using CNNs that have been pre-trained to accurately categorize natural object stimuli 9,10 .…”