SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks

Huang, Xun; Shen, Chengyao; Boix, Xavier; Zhao, Qi

doi:10.1109/iccv.2015.38

Cited by 510 publications

(437 citation statements)

References 29 publications

Supporting

Mentioning

421

Contrasting

Unclassified

Order By: Relevance

“…Saliency models have been developed for visual attention modeling [9], [26], [27], [28] and salient object detection [29], [17], [30]. The former task aims to predict human fixation locations on natural images, while the proposed method aims to compute the pixel-wise saliency values for capturing the regions of the salient objects.…”

Section: A Saliency Detection Methodsmentioning

confidence: 99%

DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection

Li-ming

Wei

et al. 2016

IEEE Trans. on Image Process.

478

320

View full text Add to dashboard Cite

A key problem in salient object detection is how to effectively model the semantic properties of salient objects in a data-driven manner. In this paper, we propose a multi-task deep saliency model based on a fully convolutional neural network with global input (whole raw images) and global output (whole saliency maps). In principle, the proposed saliency model takes a data-driven strategy for encoding the underlying saliency prior information, and then sets up a multi-task learning scheme for exploring the intrinsic correlations between saliency detection and semantic image segmentation. Through collaborative feature learning from such two correlated tasks, the shared fully convolutional layers produce effective features for object perception. Moreover, it is capable of capturing the semantic information on salient objects across different levels using the fully convolutional layers, which investigate the feature-sharing properties of salient object detection with a great reduction of feature redundancy. Finally, we present a graph Laplacian regularized nonlinear regression model for saliency refinement. Experimental results demonstrate the effectiveness of our approach in comparison with the state-of-the-art approaches.

show abstract

Section: A Saliency Detection Methodsmentioning

confidence: 99%

DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection

Li-ming

Wei

et al. 2016

IEEE Trans. on Image Process.

478

320

View full text Add to dashboard Cite

show abstract

“…The saliency loss L S is defined as the pixel-level content loss between Â and A , which is used to train the generator of the attention maps. Two loss functions were experimented with in the models shown in Table 1: M-SEN MSE uses the mean squared error (MSE) loss, a base-line loss as it has been used in many visual saliency prediction works [11]; M-SEN BCE uses binary cross-entropy (BCE) loss, which is mathematically equivalent to Kullback-Leibler divergence, arguably the best metric to measure saliency prediction performance [12]. For the classification task, cross-entropy loss was used as L C , the same as in [5].…”

Section: Methodsmentioning

confidence: 99%

Multi-task SonoEyeNet: Detection of Fetal Standardized Planes Assisted by Generated Sonographer Attention Maps

Cai

Sharma

Chatelain

et al. 2018

Medical Image Computing and Computer Assisted Intervention – MICCAI 2018

View full text Add to dashboard Cite

We present a novel multi-task convolutional neural network called Multi-task SonoEyeNet (M-SEN) that learns to generate clinically relevant visual attention maps using sonographer gaze tracking data on input ultrasound (US) video frames so as to assist standardized abdominal circumference (AC) plane detection. Our architecture consists of a generator and a discriminator, which are trained in an adversarial scheme. The generator learns sonographer attention on a given US video frame to predict the frame label (standardized AC plane / background). The discriminator further fine-tunes the predicted attention map by encouraging it to mimick the ground-truth sonographer attention map. The novel model expands the potential clinical usefulness of a previous model by eliminating the requirement of input gaze tracking data during inference without compromising its plane detection performance (Precision: 96.8, Recall: 96.2, F-1 score: 96.5).

show abstract

“…We constructed a ground truth saliency mask G , which consisted of binary values corresponding to the size of glimpse g t , with elements in G equal to one at location where the bounding box overlaps with g t and equal to zero otherwise. The Kullback–Leibler (KL) divergence was utilized to update the parameters of L sal since the KL divergence was shown to be more effective than other metrics in training networks to predict saliency . Regarding the saliency map as a probability distribution of saliency, we can compute the KL divergence L sal between the predicted saliency mask γ t and the ground truth distribution G as follows:

L_{sal} = \sum bold-italicG l o g \frac{G}{γ_{bold-italict}},

where L sal is to be minimized.…”

Section: Methodsmentioning

confidence: 99%

Attention‐aware fully convolutional neural network with convolutional long short‐term memory network for ultrasound‐based motion tracking

Huang

Lü

et al. 2019

Medical Physics

View full text Add to dashboard Cite

Purpose One of the promising options for motion management in radiation therapy (RT) is the use of LINAC‐compatible robotic‐arm‐mounted ultrasound imaging system due to its high soft tissue contrast, real‐time capability, absence of ionizing radiation, and low cost. The purpose of this work is to develop a novel deep learning‐based real‐time motion tracking strategy for ultrasound image‐guided RT. Methods The proposed tracker combined the attention‐aware fully convolutional neural network (FCNN) and the convolutional long short‐term memory network (CLSTM) that is end‐to‐end trainable. The glimpse sensor module was built inside the attention‐aware FCNN to discard majority of background by focusing on a region containing the object of interest. FCNN extracted discriminating spatial features of glimpse to facilitate temporal modeling for CLSTM. The saliency mask computed from CLSTM refined the features particular to the tracked landmarks. Moreover, the multitask loss strategy including bounding box loss, localization loss, saliency loss, and adaptive loss weighting term was utilized to facilitate training convergence and avoid over/underfitting. The tracker was tested on the databases provided by MICCAI 2015 challenges, and the ground truth data were obtained with the help of brute force‐based template matching technology. Results The mean tracking error of 0.97 ± 0.52 mm and maximum tracking error of 1.94 mm were observed for 85 point landmarks across 39 ultrasound cases compared to the ground truth annotations. The tracking speed per frame per landmark with the GPU implementation ranged from 66 and 101 frames per second, which largely exceeded the ultrasound imaging rate. Conclusion The results demonstrated the robustness and accuracy of the proposed deep learning‐based motion estimation, despite the existence of some known shortcomings of ultrasound imaging such as speckle noise. The tracking speed of the system was found to be remarkable, sufficiently fast for real‐time applications in RT environment. The approach provides a valuable tool to guide RT treatment with beam gating or multileaf collimator (MLC) tracking in real time.

show abstract

SALICON: Reducing the Semantic Gap in Saliency Prediction by Adapting Deep Neural Networks

Cited by 510 publications

References 29 publications

DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection

DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection

Multi-task SonoEyeNet: Detection of Fetal Standardized Planes Assisted by Generated Sonographer Attention Maps

Attention‐aware fully convolutional neural network with convolutional long short‐term memory network for ultrasound‐based motion tracking

Contact Info

Product

Resources

About