SalSAC: A Video Saliency Prediction Model with Shuffled Attentions and Correlation-Based ConvLSTM

Abstract

The performance of predicting human fixations in videos has been much enhanced with the help of development of the convolutional neural networks (CNN). In this paper, we propose a novel end-to-end neural network “SalSAC” for video saliency prediction, which uses the CNN-LSTM-Attention as the basic architecture and utilizes the information from both static and dynamic aspects. To better represent the static information of each frame, we first extract multi-level features of same size from different layers of the encoder CNN and calculate the corresponding multi-level attentions, then we randomly shuffle these attention maps among levels and multiply them to the extracted multi-level features respectively. Through this way, we leverage the attention consistency across different layers to improve the robustness of the network. On the dynamic aspect, we propose a correlation-based ConvLSTM to appropriately balance the influence of the current and preceding frames to the prediction. Experimental results on the DHF1K, Hollywood2 and UCF-sports datasets show that SalSAC outperforms many existing state-of-the-art methods.

A Novel Spatio-Temporal 3D Convolutional Encoder-Decoder Network for Dynamic Saliency Prediction

“…STRA-Net [11] adopts a dual-pathway architecture combining 2D ResNet50 with ConvLSTM, while the proposed STCED utilizes a dualpathway 3D ResNet50 as the encoder, which implicitly justifies the capability of 3DCNN. As shown in Table 3, all of these four models [11], [13]- [15] perform far behind STCED on the DHF1K test set, which verifies the effectiveness of the proposed model. 1 https://mmcheng.net/videosal/…”

“…Several models [7], [10], [13], [14] based on LSTM have only one data stream. In ACLNet [7], spatial features of each frame are extracted by a CNN with attention subnetwork.…”

“…Linardos et al [10] estimate dynamic saliency by adopting both convolutional LSTM (ConvLSTM) and exponential moving average (EMA) [37] as the temporal modeling module, which is inserted into a VGG16 [38] based encoder-decoder network. Based on a VGG encoder-decoder backbone, SalSAC [13] proposes a shuffled attention module and correlation-based ConvLSTM to improve temporal modeling for dynamic saliency prediction. Chen et al [14] develop a spatiotemporal feature alignment network, which contains a multi-scale deformable convolutional alignment subnetwork and a bidirectional Bi-ConvLSTM subnetwork.…”

“…Several sequences are illustrated with detailed analysis to show that STCED performs better than other models on two challenging cases, which are saliency allocation and shifting saliency focus when there are multiple instances in the scene. [13], TASED-Net [16], STRA-Net [11], SalEMA [10], ACLNet [7], DeepVS [9], STSConvNet [8], and models proposed by Sun et al [17], Chen et al [14], and Zhang et al [15]. For a fair comparison, all metrics are evaluated on the private test set of DHF1K and obtain from the benchmark website 1 .…”

“…The four models of STRA-Net [11], SalSAC [13], Chen et al [14], and Zhang et al [15] are based on architectures combining 2D CNN with ConvLSTM. STRA-Net [11] adopts a dual-pathway architecture combining 2D ResNet50 with ConvLSTM, while the proposed STCED utilizes a dualpathway 3D ResNet50 as the encoder, which implicitly justifies the capability of 3DCNN.…”