Video Saliency Prediction Using Spatiotemporal Residual Attentive Networks

Lai, Qiuxia; Yang, Yi; Sun, Hao; Shen, Jianbing

doi:10.1109/tip.2019.2936112

Cited by 123 publications

(74 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[6] used attention mechanisms to encode static saliency information and apply LSTMs to learn temporal saliency representation across consecutive video frames. [3] put forward a composite attention mechanism that learned multi-scale local attentions and global attention priors for enhancing spatio-temporal features. Fig.…”

Section: Discussionmentioning

confidence: 99%

Co-Saliency Detection With Co-Attention Fully Convolutional Network

Gao

Zhao²,

Liu

et al. 2021

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Co-saliency detection aims to detect common salient objects from a group of relevant images. Some attempts have been made with the Fully Convolutional Network (FCN) framework and achieve satisfactory detection results. However, due to stacking convolution layers and pooling operation, the boundary details tend to be lost. In addition, existing models often utilize the extracted features without discrimination, leading to redundancy in representation since actually not all features are helpful to the final prediction and some even bring distraction. In this paper, we propose a co-attention module embedded FCN framework, called as Co-Attention FCN (CA-FCN). Specifically, the co-attention module is plugged into the high-level convolution layers of FCN, which can assign larger attention weights on the common salient objects and smaller ones on the background and uncommon distractors to boost final detection performance. Extensive experiments on three popular co-saliency benchmark datasets demonstrate the superiority of the proposed CA-FCN, which outperforms state-of-the-arts in most cases. Besides, the effectiveness of our new co-attention module is also validated with ablation studies.

show abstract

Section: Discussionmentioning

confidence: 99%

Co-Saliency Detection With Co-Attention Fully Convolutional Network

Gao

Zhao²,

Liu

et al. 2021

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…The OM-CNN consists of two branches: one extracts spatial features of objects based on YOLO [39], the other extracts motion information with a variation of FlowNet [40]. Lai et al [11] propose a residual attention network to predict video saliency. The network uses two tightly coupled streams to extract appearance and motion features, and then a lightweight ConvGRU, an alternative to ConvLSTM, to model long-term temporal dependence.…”

Section: B Modern Dynamic Saliency Modelsmentioning

confidence: 99%

A Novel Spatio-Temporal 3D Convolutional Encoder-Decoder Network for Dynamic Saliency Prediction

Shi

2021

IEEE Access

View full text Add to dashboard Cite

As human beings are living in an always changing environment, predicting saliency maps from dynamic visual stimulus is of importance for modeling human visual system. Compared with human behavior, recent models based on LSTM and 3DCNN are still not good enough due to the limitation in spatio-temporal feature representation. In this paper, a novel 3D convolutional encoder-decoder architecture is proposed for saliency prediction on dynamic scenes. The encoder consists of two subnetworks to extract both spatial and temporal features in parallel with intermediate fusion, respectively. The saliency map is produced in decoder by firstly enlarging features in spatial dimensions and then aggregating temporal information. Specially designed structures can transfer pooling indices from encoder to decoder, which helps the generation of location-aware saliency maps. The proposed network can be trained and inferred in an end-to-end manner. Experimental results on benchmark DHF1K show that the proposed model achieves the state-of-the-art performance on key metrics including both normalized scanpath saliency and Pearson's correlation coefficient.

show abstract

“…Additionally, structure priors in images can be efficiently fused using multi‐modal inputs. For example, spatial ranking maps are supervised by panoptic segmentation labels to alleviate overlapping problems among various classes [42], high‐resolution feature maps are refined with inputs from multiple paths [49], streams, respectively, given stacked optical flows and colour images model motions and appreances [50], and gated convolutional layers enforce boundary information merely processed in a shape stream [32]; images in axial, coronal, and sagittal views are independently segmented and fused into a single result with union operations [43].…”

Section: Related Workmentioning

confidence: 99%