Revisiting Video Saliency: A Large-Scale Benchmark and a New Model

Abstract

In this work, we contribute to video saliency research in two ways. First, we introduce a new benchmark for predicting human eye movements during dynamic scene freeviewing, which is long-time urged in this field. Our dataset, named DHF1K (Dynamic Human Fixation), consists of 1K high-quality, elaborately selected video sequences spanning a large range of scenes, motions, object types and background complexity. Existing video saliency datasets lack variety and generality of common dynamic scenes and fall short in covering challenging situations in unconstrained environments. In contrast, DHF1K makes a significant leap in terms of scalability, diversity and difficulty, and is expected to boost video saliency modeling. Second, we propose a novel video saliency model that augments the CNN-LSTM network architecture with an attention mechanism to enable fast, end-to-end saliency learning. The attention mechanism explicitly encodes static saliency information, thus allowing LSTM to focus on learning more flexible temporal saliency representation across successive frames. Such a design fully leverages existing large-scale static fixation datasets, avoids overfitting, and significantly improves training efficiency and testing performance. We thoroughly examine the performance of our model, with respect to state-of-the-art saliency models, on three largescale datasets (i.e., DHF1K, Hollywood2, UCF sports). Experimental results over more than 1.2K testing videos containing 400K frames demonstrate that our model outperforms other competitors.

TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection

“…Metric NSS CC SIM AUC-J s-AUC GBVS 1.775 0.331 0.201 0.855 0.592 SALICON [20] 1.901 0.327 0.232 0.857 0.590 OM-CNN [19] 1.911 0.344 0.256 0.856 0.583 DVA [38] 2.013 0.358 0.262 0.860 0.595 SalGAN [29] 2.043 0.370 0.262 0.866 0.709 ACLNet [39] 2.354 0.434 0.315 0.890 0.601 TASED-Net 2.667 0.470 0.361 0.895 0.712 itative results of our model and ACLNet for the better and worse cases are given in Figure 5 (see Supplementary material for more examples of qualitative results). As shown in (a) and (b) in Figure 5, TASED-Net seems highly sensitive to salient moving objects and less sensitive to background objects, which is consistent with the goal of video saliency in general.…”

“…Since the ground-truth annotations for the test set of DHF1K [39] are hidden for fair comparison, we first evaluate variants of our model on the validation set. The performance of TASED-Net with different T and temporal aggregation strategies are compared in Table 1.…”

“…The results indicate that TASED-Net with T = 32 and late two-step aggregation performs the best since this configuration achieves the best performance across most metrics (it has 21.2M Params and 63.2G FLOPs; more results on different T 's are provided in Section 4.5). We believe that late two-step aggregation performs better than early two-step aggregation because the feature maps used in spatial upscaling have a Table 1: Performance comparison of TASED-Net with different T s (shown in parentheses) and temporal aggregation strategies on the validation set of DHF1K [39]. The late two-step approach performs the best since it utilizes temporally rich features while avoiding overfitting.…”

“…For the rest of the paper, we report the performance of TASED-Net with T = 32, late two-step aggregation, and pre-training. Next, we submitted our results to the DHF1K online benchmark [39]. The performance of TASED-Net and previous state-of-the-art methods on the test set of DHF1K is reported in Table 2.…”

“…Our model outperforms other methods by a wide margin across all evaluation metrics. We note that ACLNet [39], the leading state-of-the-art method, is arguably better-primed for saliency detection than TASED-Net-it has a component pre-trained on an image-saliency dataset, SALICON [20], whereas we pre-train the encoder network of TASED-Net on an action recognition dataset. The higher performance of TASED-Net suggests that pretraining on a large-scale video dataset plays a significant role in performing well on other tasks in general.…”