Contextual encoder–decoder network for visual saliency prediction

Kröner, Alexander; Senden, Mario; Driessens, Kurt; Goebel, Rainer

doi:10.1016/j.neunet.2020.05.004

Cited by 155 publications

(92 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…All three models did comparable jobs of predicting whether a scene region would be fixated or not (MSI-Net=0.82, DeepGaze II=0.83, , SAM-ResNet=0.81). Taken together our results replicate previous findings [36][37][38] and establish that MSI-Net, DeepGaze II, and SAM-ResNet also predict scene attention well in active viewing tasks.…”

Section: Resultssupporting

confidence: 90%

See 1 more Smart Citation

What do deep saliency models learn about where we look in scenes?

Hayes

Henderson

2021

Preprint

View full text Add to dashboard Cite

Deep saliency models represent the current state-of-the-art for predicting where humans look in real-world scenes. However, for deep saliency models to inform cognitive theories of attention, we need to know how deep saliency models predict where people look. Here we open the black box of deep saliency models using an approach that models the association between the output of 3 prominent deep saliency models (MSI-Net, DeepGaze II, and SAM-ResNet) and low-, mid-, and high-level scene features. Specifically, we measured the association between each deep saliency model and low-level image saliency, mid-level contour symmetry and junctions, and high-level meaning by applying a mixed effects modeling approach to a large eye movement dataset. We found that despite different architectures, training regimens, and loss functions, all three deep saliency models were most strongly associated with high-level meaning. These findings suggest that deep saliency models are primarily learning image features associated with scene meaning.

show abstract

Section: Resultssupporting

confidence: 90%

“…We compared 3 of the best performing deep saliency models on the MIT saliency benchmark 1 : the multi-scale information network (MSI-Net) 36 , DeepGaze II 37 , and the saliency attentive model (SAM-ResNet) 38 . Each deep saliency model takes an image as input and produces a predicted saliency map as output.…”

Section: Deep Saliency Modelsmentioning

confidence: 99%

What do deep saliency models learn about where we look in scenes?

Hayes

Henderson

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Along the same lines, a recently developed network has accurately predicted visual saliency using a similar encoderdecoder network ([Kroner et al, 2020]). Since we know the human visual system's coding of saliency must be robust to occlusion and clutter, it would be interesting to compare network objectives of predicting occluded visual features and predicting saliency.…”

Section: Future Directionsmentioning

confidence: 95%

A Self-Supervised Deep Neural Network for Image Completion Resembles Early Visual Cortex fMRI Activity Patterns for Occluded Scenes

Svanera

Morgan

Petro

et al. 2020

Preprint

View full text Add to dashboard Cite

The promise of artificial intelligence in understanding biological vision relies on the comparison of computational models with brain data that captures functional principles of visual information processing. Deep neural networks (DNN) have successfully matched the transformations in hierarchical processing occurring along the brain's feedforward visual pathway extending into ventral temporal cortex. However, we are still to learn if DNNs can successfully describe feedback processes in early visual cortex. Here, we investigated similarities between human early visual cortex and a DNN with encoder/decoder architecture, trained in an unsupervised fashion to fill occlusions and reconstruct an unseen image. Using Representational Similarity Analysis (RSA), we compared 3T fMRI data from a non-stimulated patch of early visual cortex in human participants viewing partially occluded images, with the different DNN layer activations from the same images. Results show that our network outperforms a classical supervised network (VGG16) in terms of similarity to fMRI data, meaning that improved neural network models of vision need to incorporate architectures capturing cortical feedback processing. We also find that DNN decoder pathway activations are more similar to brain processing compared to encoder activations, suggesting an integration of mid-and low/middle-level features in early visual cortex. Challenging an AI model and the human brain to solve the same task offers a valuable way to compare DNNs with brain data and helps to constrain our understanding of information processing such as neuronal predictive coding.Keywords Unsupervised deep neural networks · Encoder/decoder architecture · 3T fMRI · Early visual cortex · Representational Similarity Analysis (RSA).

show abstract

“…Then, we describe how the models interact with each other. The visual system is composed of the segmentation model of Francis et al (2017), a retina model inspired by Ambrosano et al (2016), and a saliency model, which is a simplified version of the model introduced by Kroner et al (2019). These specific parts of human vision were chosen because the segmentation model already explains many features of visual crowding (Figure 4) and because retinal processing, as well as saliency computation, are potential sources of anisotropy for the segmentation output.…”

Section: Methodsmentioning

confidence: 99%

“…The model is an encoder-decoder network that learned a non-linear mapping from raw images to topographic fixation maps. It constitutes a simplified version of the model introduced by Kroner et al (2019), pruning the contextual layers to achieve computationally more efficient image processing. The VGG16 architecture (Simonyan and Zisserman, 2014), pre-trained on a visual classification task, serves as the model backbone to detect high-level features in the input space.…”

Section: Methodsmentioning

confidence: 99%

Running Large-Scale Simulations on the Neurorobotics Platform to Understand Vision – The Case of Visual Crowding

et al. 2019

Self Cite

View full text Add to dashboard Cite

Traditionally, human vision research has focused on specific paradigms and proposed models to explain very specific properties of visual perception. However, the complexity and scope of modern psychophysical paradigms undermine the success of this approach. For example, perception of an element strongly deteriorates when neighboring elements are presented in addition (visual crowding). As it was shown recently, the magnitude of deterioration depends not only on the directly neighboring elements but on almost all elements and their specific configuration. Hence, to fully explain human visual perception, one needs to take large parts of the visual field into account and combine all the aspects of vision that become relevant at such scale. These efforts require sophisticated and collaborative modeling. The Neurorobotics Platform (NRP) of the Human Brain Project offers a unique opportunity to connect models of all sorts of visual functions, even those developed by different research groups, into a coherently functioning system. Here, we describe how we used the NRP to connect and simulate a segmentation model, a retina model, and a saliency model to explain complex results about visual perception. The combination of models highlights the versatility of the NRP and provides novel explanations for inward-outward anisotropy in visual crowding.

show abstract

Contextual encoder–decoder network for visual saliency prediction

Cited by 155 publications

References 37 publications

What do deep saliency models learn about where we look in scenes?

What do deep saliency models learn about where we look in scenes?

A Self-Supervised Deep Neural Network for Image Completion Resembles Early Visual Cortex fMRI Activity Patterns for Occluded Scenes

Running Large-Scale Simulations on the Neurorobotics Platform to Understand Vision – The Case of Visual Crowding

Contact Info

Product

Resources

About