DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling

Linardos, Akis; Kümmerer, Matthias; Press, Ori; Bethge, Matthias

doi:10.1109/iccv48922.2021.01268

Cited by 45 publications

(34 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In DeepGaze II [45], the fine-tuning of the VGG-19 features is used for saliency detection. The performance of saliency model is enhanced by using ResNet50 features instead of the VGG19 of DeepGaze II in [46]. In addition to the aforementioned algorithms which are mostly in the spatial domain, some models work in the frequency domain and benefit from Fourier transform, discrete cosine transform, etc.…”

Section: Related Workmentioning

confidence: 99%

A Dynamic Bottom-Up Saliency Detection Method for Still Images

Sadeghi

Kamkar

Moghaddam

2022

Preprint

View full text Add to dashboard Cite

Introduction: Existing saliency detection algorithms in the literature have ignored the importance of time. They create a static saliency map for the whole recording time. However, bottom-up and top-down attention continuously compete and the salient regions change through time. In this paper, we propose an unsupervised algorithm to predict the dynamic evolution of bottom-up saliency in images. Method: We compute the variation of low-level features within non-overlapping patches of the input image. A patch with higher variation is considered more salient. We use a threshold to ignore less salient parts and create a map. A weighted sum of this map and its center of mass is calculated to provide the saliency map. The threshold and weights are set dynamically. We use the MIT1003 and DOVES datasets for evaluation and break the recording to multiple 100ms or 500ms-time intervals. A separate ground-truth is created for each interval. Then, the predicted dynamic saliency map is compared to the ground-truth using Normalized Scanpath Saliency, Kullback-Leibler divergence, Similarity, and Linear Correlation Coefficient metrics. Results: The proposed method outperformed the competitors on DOVES dataset. It also had an acceptable performance on MIT1003 especially within 0-400ms after stimulus onset. Conclusion: This dynamic algorithm can predict an image's salient regions better than the static methods as saliency detection is inherently a dynamic process. This method is biologically-plausible and in-line with the recent findings of the creation of a bottom-up saliency map in the primary visual cortex or superior colliculus.

show abstract

Section: Related Workmentioning

confidence: 99%

A Dynamic Bottom-Up Saliency Detection Method for Still Images

Sadeghi

Kamkar

Moghaddam

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…• Saliency Baseline: As a complement to the baselines above, this baseline selects the query image whose original pixels hidden by the occlusion patch have a lower probability of being looked at by humans. This simulates that participants select the image with a patch that occludes less prominent information and is estimated with the saliency prediction model DeepGaze IIE (Linardos et al, 2021). Specifically, we pass the unoccluded query image through the saliency prediction model DeepGaze IIE (Linardos et al, 2021), which yields a probability density over the entire image.…”

Section: A5 Baselinesmentioning

confidence: 99%

“…This simulates that participants select the image with a patch that occludes less prominent information and is estimated with the saliency prediction model DeepGaze IIE (Linardos et al, 2021). Specifically, we pass the unoccluded query image through the saliency prediction model DeepGaze IIE (Linardos et al, 2021), which yields a probability density over the entire image. Next, we integrate said density over each of the two square patches.…”

Section: A5 Baselinesmentioning

confidence: 99%

How Well do Feature Visualizations Support Causal Understanding of CNN Activations?

Zimmermann¹,

Borowski²,

Geirhos³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

One widely used approach towards understanding the inner workings of deep convolutional neural networks is to visualize unit responses via activation maximization. Feature visualizations via activation maximization are thought to provide humans with precise information about the image features that cause a unit to be activated. If this is indeed true, these synthetic images should enable humans to predict the effect of an intervention, such as whether occluding a certain patch of the image (say, a dog's head) changes a unit's activation. Here, we test this hypothesis by asking humans to predict which of two square occlusions causes a larger change to a unit's activation. Both a large-scale crowdsourced experiment and measurements with experts show that on average, the extremely activating feature visualizations by Olah et al. (2017) indeed help humans on this task (67 ± 4 % accuracy; baseline performance without any visualizations is 60 ± 3 %). However, they do not provide any significant advantage over other visualizations (such as e.g. dataset samples), which yield similar performance (66 ± 3 % to 67 ± 3 % accuracy). Taken together, we propose an objective psychophysical task to quantify the benefit of unit-level interpretability methods for humans, and find no evidence that feature visualizations provide humans with better "causal understanding" than simple alternative visualizations.

show abstract

“…Yang et al [43] approximated the foveated retina by using the segmentation maps of a full-resolution image and its blurred version, predicted by a pretrained Panoptic-FPN [21], to approximate high-resolution fovea and low-resolution peripheral, respectively. Like other models for predicting human attention [30,24,25,6,46], both approaches rely on pretrained networks to extract image features and train much smaller networks for the downstream tasks using transfer learning, usually due to the lack of human fixation data for training. Also noteworthy is that these approaches apply networks pretrained on full-resolution images (e.g., ResNets [14] trained on Im-ageNet [38]), expecting the pretrained networks to approximate how humans perceive blurred images.…”

Section: Introductionmentioning

confidence: 99%

Target-absent Human Attention

Yang¹,

Mondal²,

Ahn³

et al. 2022

Preprint

View full text Add to dashboard Cite

The prediction of human gaze behavior is important for building human-computer interactive systems that can anticipate a user's attention. Computer vision models have been developed to predict the fixations made by people as they search for target objects. But what about when the image has no target? Equally important is to know how people search when they cannot find a target, and when they would stop searching. In this paper, we propose the first data-driven computational model that addresses the search-termination problem and predicts the scanpath of search fixations made by people searching for targets that do not appear in images. We model visual search as an imitation learning problem and represent the internal knowledge that the viewer acquires through fixations using a novel state representation that we call Foveated Feature Maps (FFMs). FFMs integrate a simulated foveated retina into a pretrained ConvNet that produces an in-network feature pyramid, all with minimal computational overhead. Our method integrates FFMs as the state representation in inverse reinforcement learning. Experimentally, we improve the state of the art in predicting human target-absent search behavior on the COCO-Search18 dataset.

show abstract

DeepGaze IIE: Calibrated prediction in and out-of-domain for state-of-the-art saliency modeling

Cited by 45 publications

References 24 publications

A Dynamic Bottom-Up Saliency Detection Method for Still Images

A Dynamic Bottom-Up Saliency Detection Method for Still Images

How Well do Feature Visualizations Support Causal Understanding of CNN Activations?

Target-absent Human Attention

Contact Info

Product

Resources

About