A Novel Spatio-Temporal 3D Convolutional Encoder-Decoder Network for Dynamic Saliency Prediction

Abstract

As human beings are living in an always changing environment, predicting saliency maps from dynamic visual stimulus is of importance for modeling human visual system. Compared with human behavior, recent models based on LSTM and 3DCNN are still not good enough due to the limitation in spatio-temporal feature representation. In this paper, a novel 3D convolutional encoder-decoder architecture is proposed for saliency prediction on dynamic scenes. The encoder consists of two subnetworks to extract both spatial and temporal features in parallel with intermediate fusion, respectively. The saliency map is produced in decoder by firstly enlarging features in spatial dimensions and then aggregating temporal information. Specially designed structures can transfer pooling indices from encoder to decoder, which helps the generation of location-aware saliency maps. The proposed network can be trained and inferred in an end-to-end manner. Experimental results on benchmark DHF1K show that the proposed model achieves the state-of-the-art performance on key metrics including both normalized scanpath saliency and Pearson's correlation coefficient.

Video saliency detection using modified high efficiency video coding and background modelling

“…It was one of the quickest approaches up to this point. They have given us knowledge on hyper saliency [43]. Convolutional neural networks are trained using manual algorithmic annotations of smooth pursuits, and the findings are developed with the aid of 26 dynamic saliency models that are freely available online.…”

TinyHD: Efficient Video Saliency Prediction with Heterogeneous Decoders using Hierarchical Maps Distillation

“…RecSal [30] predicts multiple saliency maps in a multiobjective training framework. Recent works introduce more [4,18,22,37,42]; U-Net-like architecture, with features sharing between encoder and decoder [6,16,19,25]; Deep Layer Aggregation [39]; Hierarchical intermediate map aggregation [1,30,35].…”

“…Figure 3: A taxonomy of decoding strategies commonly employed in video saliency prediction. Subfigures(topleft, top-right, bottom-left, bottom-right): Independent encoder and decoder, with no feature sharing between the two paths[4,18,22,37,42]; U-Net-like architecture, with features sharing between encoder and decoder[6,16,19,25]; Deep Layer Aggregation[39]; Hierarchical intermediate map aggregation[1,30,35].…”

Video saliency-detection using custom spatiotemporal fusion method

“…It has a database named dynamic human fixation 1K (DHF1K) that helps in pointing out fixations that are needed during dynamic scene free viewing, then there is the attentive convolutional neural network-long short-term memory network (ACLNet) which has augmentations to the original convolutional neural network and long short-term memory (CNN-LSTM) model to enable fast end-to-end saliency learning. In this paper [21], [22] they have made some corrections in the smooth pursuits (SP) logic. It involves manual annotations of the SPs with fixation along the arithmetic points and SP salient locations by training slicing convolutional neural networks.…”