DeepGaze II: Predicting fixations from deep features over time and tasks

Kümmerer, Matthias; Wallis, T. E.; Bethge, Matthias

doi:10.1167/17.10.1147

Cited by 131 publications

(178 citation statements)

References 11 publications

Supporting

Mentioning

175

Contrasting

Order By: Relevance

“…The model used Krizhevsky network to compute filter responses and a full convolution to learn the saliency model. Further more, a probabilistic model is introduced [97], which used VGG-19 features, incorporated center bias, and used a maximum likelihood learning to train the model. 8) Other models: Several other saliency models do not fit to the previously mentioned categories.…”

Section: ) Bayesian Modelsmentioning

confidence: 99%

A Comparison Study of Saliency Models for Fixation Prediction on Infants and Adults

Mahdi

Schlesinger

et al. 2018

IEEE Trans. Cogn. Dev. Syst.

View full text Add to dashboard Cite

Abstract-Various saliency models have been developed over the years. The performance of saliency models is typically evaluated based on databases of experimentally recorded adult eye fixations. Although studies on infant gaze patterns have attracted much attention recently, saliency based models have not been widely applied for prediction of infant gaze patterns. In this study, we conduct a comprehensive comparison study of eight state-ofthe-art saliency models on predictions of experimentally captured fixations from infants and adults. Seven evaluation metrics are used to evaluate and compare the performance of saliency models. The results demonstrate a consistent performance of saliency models predicting adult fixations over infant fixations in terms of overlap, center fitting, intersection, information loss of approximation, and spatial distance between the distributions of saliency map and fixation map. In saliency and baselines models performance ranking, the results show that GBVS and Itti models are among the top three contenders, infants and adults have bias toward the centers of images, and all models and the center baseline model outperformed the chance baseline model.

show abstract

Section: ) Bayesian Modelsmentioning

confidence: 99%

A Comparison Study of Saliency Models for Fixation Prediction on Infants and Adults

Mahdi

Schlesinger

et al. 2018

IEEE Trans. Cogn. Dev. Syst.

View full text Add to dashboard Cite

show abstract

“…More recently, a plethora of deep learning based methods have been proposed for static saliency prediction. Kümmerer et al [26,27] proposed two deep saliency prediction networks, DeepGaze I and DeepGaze II, that was built on the AlexNet [25] and VGG-19 [40] models respectively. Pan et al [33] used a GAN to generate saliency maps.…”

Section: Related Workmentioning

confidence: 99%

SalGaze: Personalizing Gaze Estimation using Visual Saliency

Chang

Martino

Qiu

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Traditional gaze estimation methods typically require explicit user calibration to achieve high accuracy. This process is cumbersome and recalibration is often required when there are changes in factors such as illumination and pose. To address this challenge, we introduce SalGaze, a framework that utilizes saliency information in the visual content to transparently adapt the gaze estimation algorithm to the user without explicit user calibration. We design an algorithm to transform a saliency map into a differentiable loss map that can be used for the optimization of CNN-based models. SalGaze is also able to greatly augment standard point calibration data with implicit video saliency calibration data using a unified framework. We show accuracy improvements over 24% using our technique on existing methods.

show abstract

“…We tested how well correlated the model's success rates (averaged over n=100 repetitions) were with human success rates (averaged over n=39 participants) across all images. For this analysis, we employed a saliency map generated by the DeepGaze algorithm (Kümmerer, Wallis, and Bethge 2016), rather than the frequency-tuned salient region detection algorithm (Methods). Remarkably, the model's performance strongly correlated with human performance, across images (Fig.…”

Section: Model Performance Mimics Human Performance Quantitativelymentioning

confidence: 99%

Neurally-constrained modeling of human gaze strategies in a change blindness task

Jagatap

Jain

Purokayastha

et al. 2019

Preprint

View full text Add to dashboard Cite

Visual attention enables us to engage selectively with the most important events in the world around us. Yet, sometimes, we fail to notice salient events. "Change blindness" -the surprising inability to detect and identify salient changes that occur in flashing visual images -enables measuring such failures in a laboratory setting. We discovered that human participants (n=39) varied widely (by twofold) in their ability to detect changes when tested on a laboratory change blindness task. To understand the reasons for these differences in change detection abilities, we characterized eyemovement patterns and gaze strategies as participants scanned these images. Surprisingly, we found no systematic differences between scan paths, fixation maps or saccade patterns between participants who were successful at detecting changes, versus those who were not. Yet, two lowlevel gaze metrics -the mean fixation duration and the variance of saccade amplitudessystematically predicted change detection success. To explain the mechanism by which these gaze metrics could influence performance, we developed a neurally constrained model, based on the Bayesian framework of sequential probability ratio testing (SPRT), which simulated gaze strategies of successful and unsuccessful observers. The model's ability to detect changes varied systematically with mean fixation duration and saccade amplitude variance, closely mimicking observations in the human data. Moreover, the model's success rates correlated robustly with human observers' success rates, across images. Our model explains putative human attention mechanisms during change blindness tasks and provides key insights into effective strategies for shifting gaze and attention for artificial agents navigating dynamic, crowded environments. Author SummaryOur brain has the remarkable capacity to pay attention, selectively, to the most important events in the world around us. Yet, sometimes, we fail spectacularly to notice even the most salient events.We tested this phenomenon in the laboratory with a change-blindness experiment, by having participants freely scan and detect changes across discontinuous image pairs. Participants varied widely in their ability to detect these changes. Surprisingly, their success correlated with differences in low-level gaze metrics. A Bayesian model of eye movements, which incorporated neural constraints on stimulus encoding, could explain the reason for these differences, and closely mimicked human performance in this change blindness task. The model's gaze strategies provide relevant insights for artificial, neuromorphic agents navigating dynamic, crowded environments.

show abstract

DeepGaze II: Predicting fixations from deep features over time and tasks

Cited by 131 publications

References 11 publications

A Comparison Study of Saliency Models for Fixation Prediction on Infants and Adults

A Comparison Study of Saliency Models for Fixation Prediction on Infants and Adults

SalGaze: Personalizing Gaze Estimation using Visual Saliency

Neurally-constrained modeling of human gaze strategies in a change blindness task

Contact Info

Product

Resources

About