Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images

Vig, Eleonora; Dörr, Michael; Cox, David D.

doi:10.1109/cvpr.2014.358

Cited by 364 publications

(236 citation statements)

References 20 publications

(48 reference statements)

Supporting

Mentioning

234

Contrasting

Unclassified

Order By: Relevance

“…The included models are Itti et al (1) [here, two implementations have been used: one from the Saliency Toolbox and the variant specified in the graph-based visual saliency (GBVS) paper], Torralba et al (26), GBVS (27), saliency using natural statistics (SUN) (28) (for "SUN, original" we used a scale parameter of 0.64, corresponding to the pixel size of 2.3′ of visual angle of the dataset used to learn the filters; for "SUN, optimal" we did a grid search over the scale parameter; this resulted in a scale parameter of 0.15), Kienzle et al (24,29) (patch size 195 pixels corresponding to their reported optimal patch size of 5.4°). Hou and Zhang (30), AIM (31), Judd et al (23), context-aware saliency (32,33), visual saliency estimation by nonlinearly integrating features using region covariances (CovSal) (34), multiscale rarity-based saliency detection (RARE2012) (35), BMS (5,36), and finally eDN (37). Table S2 specifies the source code used for each model.…”

Section: Methodsmentioning

confidence: 99%

Information-theoretic model comparison unifies saliency metrics

Kümmerer

Wallis

Bethge

2015

Proc. Natl. Acad. Sci. U.S.A.

142

167

View full text Add to dashboard Cite

Learning the properties of an image associated with human gaze placement is important both for understanding how biological systems explore the environment and for computer vision applications. There is a large literature on quantitative eye movement models that seeks to predict fixations from images (sometimes termed "saliency" prediction). A major problem known to the field is that existing model comparison metrics give inconsistent results, causing confusion. We argue that the primary reason for these inconsistencies is because different metrics and models use different definitions of what a "saliency map" entails. For example, some metrics expect a model to account for image-independent central fixation bias whereas others will penalize a model that does. Here we bring saliency evaluation into the domain of information by framing fixation prediction models probabilistically and calculating information gain. We jointly optimize the scale, the center bias, and spatial blurring of all models within this framework. Evaluating existing metrics on these rephrased models produces almost perfect agreement in model rankings across the metrics. Model performance is separated from center bias and spatial blurring, avoiding the confounding of these factors in model comparison. We additionally provide a method to show where and how models fail to capture information in the fixations on the pixel level. These methods are readily extended to spatiotemporal models of fixation scanpaths, and we provide a software package to facilitate their use.visual attention | eye movements | probabilistic modeling | likelihood | point processes H umans move their eyes about three times/s when exploring the environment, fixating areas of interest with the highresolution fovea. How do we determine where to fixate to learn about the scene in front of us? This question has been studied extensively from the perspective of "bottom-up" attentional guidance (1), often in a "free-viewing" task in which a human observer explores a static image for some seconds while his or her eye positions are recorded (Fig. 1A). Eye movement prediction is also applied in domains from advertising to efficient object recognition. In computer vision the problem of predicting fixations from images is often referred to as "saliency prediction," while to others "saliency" refers explicitly to some set of low-level image features (such as edges or contrast). In this paper we are concerned with predicting fixations from images, taking no position on whether the features that guide eye movements are "low" or "high" level.The field of eye movement prediction is quite mature: Beginning with the influential model of Itti et al. (1), there are now over 50 quantitative fixation prediction models, including around 10 models that seek to incorporate "top-down" effects (see refs. 2-4 for recent reviews and analyses of this extensive literature). Many of these models are designed to be biologically plausible whereas others aim purely at prediction (e.g., ref. 5). Progress is meas...

show abstract

Section: Methodsmentioning

confidence: 99%

Information-theoretic model comparison unifies saliency metrics

Kümmerer

Wallis

Bethge

2015

Proc. Natl. Acad. Sci. U.S.A.

142

167

View full text Add to dashboard Cite

show abstract

“…Such models dramatically improved the visual attention. Vig et al [95] proposed the first deep learning saliency model, which incorporated biologically inspired features, and used the standard learning pipeline. Kummerer et al [96] presented a novel way to reuse existing object recognition neural networks for fixation prediction.…”

Section: ) Bayesian Modelsmentioning

confidence: 99%

A Comparison Study of Saliency Models for Fixation Prediction on Infants and Adults

Mahdi

Schlesinger

et al. 2018

IEEE Trans. Cogn. Dev. Syst.

View full text Add to dashboard Cite

Abstract-Various saliency models have been developed over the years. The performance of saliency models is typically evaluated based on databases of experimentally recorded adult eye fixations. Although studies on infant gaze patterns have attracted much attention recently, saliency based models have not been widely applied for prediction of infant gaze patterns. In this study, we conduct a comprehensive comparison study of eight state-ofthe-art saliency models on predictions of experimentally captured fixations from infants and adults. Seven evaluation metrics are used to evaluate and compare the performance of saliency models. The results demonstrate a consistent performance of saliency models predicting adult fixations over infant fixations in terms of overlap, center fitting, intersection, information loss of approximation, and spatial distance between the distributions of saliency map and fixation map. In saliency and baselines models performance ranking, the results show that GBVS and Itti models are among the top three contenders, infants and adults have bias toward the centers of images, and all models and the center baseline model outperformed the chance baseline model.

show abstract

“…One of the first attempts to leverage deep learning for saliency prediction was Vig et al [14], using convnet layers as feature maps to classify fixated local regions. Kümmerer et al [15] introduced the model DeepGaze, built on top of the AlexNet image classification network [16].…”

Section: Related Workmentioning

confidence: 99%

Where Should Saliency Models Look Next?

Bylinskii

Recasens

Borji

et al. 2016

Lecture Notes in Computer Science

169

197

View full text Add to dashboard Cite

Abstract. Recently, large breakthroughs have been observed in saliency modeling. The top scores on saliency benchmarks have become dominated by neural network models of saliency, and some evaluation scores have begun to saturate. Large jumps in performance relative to previous models can be found across datasets, image types, and evaluation metrics. Have saliency models begun to converge on human performance? In this paper, we re-examine the current state-of-the-art using a finegrained analysis on image types, individual images, and image regions. Using experiments to gather annotations for high-density regions of human eye fixations on images in two established saliency datasets, MIT300 and CAT2000, we quantify up to 60% of the remaining errors of saliency models. We argue that to continue to approach human-level performance, saliency models will need to discover higher-level concepts in images: text, objects of gaze and action, locations of motion, and expected locations of people in images. Moreover, they will need to reason about the relative importance of image regions, such as focusing on the most important person in the room or the most informative sign on the road. More accurately tracking performance will require finer-grained evaluations and metrics. Pushing performance further will require higher-level image understanding.

show abstract

Large-Scale Optimization of Hierarchical Features for Saliency Prediction in Natural Images

Cited by 364 publications

References 20 publications

Information-theoretic model comparison unifies saliency metrics

Information-theoretic model comparison unifies saliency metrics

A Comparison Study of Saliency Models for Fixation Prediction on Infants and Adults

Where Should Saliency Models Look Next?

Contact Info

Product

Resources

About