Revisiting Video Saliency Prediction in the Deep Learning Era

Abstract

Predicting where people look in static scenes, a.k.a visual saliency, has received significant research interest recently. However, relatively less effort has been spent in understanding and modeling visual attention over dynamic scenes. This work makes three contributions to video saliency research. First, we introduce a new benchmark, called DHF1K (Dynamic Human Fixation 1K), for predicting fixations during dynamic scene free-viewing, which is a long-time need in this field. DHF1K consists of 1K high-quality elaborately-selected video sequences annotated by 17 observers using an eye tracker device. The videos span a wide range of scenes, motions, object types and backgrounds. Second, we propose a novel video saliency model, called ACLNet (Attentive CNN-LSTM Network), that augments the CNN-LSTM architecture with a supervised attention mechanism to enable fast end-to-end saliency learning. The attention mechanism explicitly encodes static saliency information, thus allowing LSTM to focus on learning a more flexible temporal saliency representation across successive frames. Such a design fully leverages existing large-scale static fixation datasets, avoids overfitting, and significantly improves training efficiency and testing performance. Third, we perform an extensive evaluation of the state-of-the-art saliency models on three datasets : DHF1K, Hollywood-2, and UCF sports. An attribute-based analysis of previous saliency models and cross-dataset generalization are also presented. Experimental results over more than 1.2K testing videos containing 400K frames demonstrate that ACLNet outperforms other contenders and has a fast processing speed (40fps using a single GPU). Our code and all the results are available at https://github.com/wenguanwang/DHF1K.

EyeTrackUAV2: A Large-Scale Binocular Eye-Tracking Dataset for UAV Videos

“…Accordingly, we defined two visual tasks participants had to perform: the first condition was an FV task while the second relates to a surveillance viewing task (Task). The former task is rather common in eye-tracking tests [32,37,43,45,50,71]. Observers were simply asked to observe visual video stimuli without performing any task.…”

“…While it is now rather easy to find eye tracking data on typical images [35,[37][38][39][40][41][42][43][44][45] or videos [46][47][48][49][50], and that there are many UAV content datasets [7,[51][52][53][54][55][56][57][58][59][60][61][62], it turns out to be extremely difficult to find eye-tracking data on UAV content. This is even truer when we consider dynamic salience, which refers to salience for video content.…”

Low-cost intelligent surveillance system based on fast CNN

“…In a technical context, a pattern can involve repeating sequences of data with time, and patterns can be utilized to predict trends and specific featural configurations in images to recognize objects. Many recognition approaches involving the use of the support vector machine (SVM) (Junoh et al, 2012), artificial neural network (ANN) (Petrosino & Maddalena, 2012), deep learning (Wang et al, 2019), and other rule-based classification systems have been developed. Performing classification using an ANN is a supervised practical strategy that has achieved satisfactory results in many classification tasks.…”

Abrupt darkening under high dynamic range (HDR) luminance invokes facilitation for high-contrast targets and grouping by luminance similarity

“…Rather than simply remapping the color histogram or normalizing an image for nearby luminance, automatic mechanisms are thought to depend on factors such as co-orientation, co-linearity, co-circularity, co-planarity, junctions, feature grouping, and transparency issues, such as smoke and rain ( Adelson, 2000 ; Anderson, 1997 ; Li, Song, Xu, Hu, Roe, & Li, 2019 ; Zemach & Rudd, 2007 ; Zucker, David, Dobbins, & Iverson, 1988 ). Biologically driven models are increasingly capable of explaining visual illusions ( Blakeslee & McCourt, 2004 ; Li, 2011 ) and predicting gaze patterns based on saliency ( Kümmerer, Wallis, Gatys, & Bethge, 2017 ; Wang, Shen, Xie, Cheng, Ling, & Borji, 2019 ), but they are data-limited to SDR images and require HDR experimentation to extend their generalizability to real-world vision.…”