Weakly Supervised Salient Object Detection With Spatiotemporal Cascade Neural Networks

Abstract

Recently, deep learning techniques have substantially boosted the performance of salient object detection in still images. However, the salient object detection in videos by using traditional handcrafted features or deep learning features is not fully investigated, probably due to the lack of sufficient manually labeled video data for saliency modeling, especially for the data-driven deep learning. This paper proposes a novel weakly supervised approach to salient object detection in a video, which can learn a robust saliency prediction model by using very limited manually labeled data and a large amount of weakly labeled data that could be easily generated in a supervised approach. Furthermore, we propose a spatiotemporal cascade neural network (SCNN) architecture for saliency modeling, in which two fully convolutional networks are cascaded to evaluate visual saliency from both spatial and temporal cues to lead the optimal video saliency prediction. The proposed approach is extensively evaluated on the widely used challenging datasets, and the experiments demonstrate that our proposed approach substantially outperforms the state-of-the-art salient object detection models. Index Terms-Video saliency, weakly supervised learning, spatiotemporal prior fusion, cascade fully convolutional network I. INTRODUCTION S ALIENT object detection, which aims to identify the objects or regions that are noticeable and mostly attract human attention in an image/video, has become a research focus of computer vision for decades. It is generally as a preprocessing step to support high-level computer vision tasks, such as object segmentation, object recognition, object tracking and content-based video compression. A number of approaches have been proposed to detect salient objects. The recent approaches based on deep Convolutional Neural Networks (CNNs), e.g., [1]-[3], have substantially improved

Overview of deep-learning based methods for salient object detection in videos

“…According to the common concepts used in deep learning methods, firstly, the global framework for each method is described in 2.1, then the deep network in each method is analyzed in 2.2, and finally an overview of the categorization of methods is shown at a functional level in 2.3. As a matter of convenience, the describled methods, are denoted as SCOMd [12], NRF [13], DHSNet [14], OSVOS [15], NLDF [16], LMP [17], SFCN [18], SegFlow [19], LVO [20], WSS [21], SCNN [22], DSS [23], SPD [24], AFNet [25] and CPD [26].…”

“…Moving objects attract large attention and thus can be regarded as salient objects in videos. As in the methods[12,22,18], we also use the 30 test videos with the provided ground truth. The DAVIS 2016 dataset[27] is a popular video dataset for video foreground segmentation.It is divided into two splits: the training (30 sequences) part used for training only and the validation (20 sequences) part for the inference.…”

Video salient object detection via spatiotemporal attention neural networks

“…Different from the cascade structure in [48], we propose a two-stream deep network to estimate salient regions. The specific framework are shown in Figure 2, which consists of four components, two network streams without shared parameters, refinement of motion information and attentive module.…”

“…In the temporal stream, we treat the RGB frames and corresponding motion priors as the network input. The temporal prior is generated by [48]. Specifically, the superpixels of the optical flow map are firstly obtained by using [10].…”

“…Among them, there are ten state-of-the-arts in video scenes and nine approaches in still images. The video salient object detection includes cluster-based co-saliency method (CS) [11], space-time saliency detection (ST) [75], segmenting saliency detection (SS) [42], saliency-aware method (SA) [59], consistent gradient based saliency (CG) [61], video salient object detection via fully convolutional networks (SFCN) [62], multi-scale spatiotemporal network (MSST) for video saliency [49], spatiotemporal cascade network (SCNN) [48], flow guided recurrent neural encoder (FGRNE) [26] and pyramid dilated deeper convlstm (PDB) [45]. The image saliency models include recurrent fully convolutional network (RFCN) [55], deep contrast learning (DCL) [27], visual saliency on multi-scale deep features (MDF) [28], deep saliency multi-task (DSMT) [30], deeply supervised salient object detection (DSS) [16], deep hierarchical saliency network (DHSN) [32], aggregating multilevel convolutional features (Amulet) [71], saliency detection with image-level supervision (WSS) [54] and learning uncertain convolutional features (UCF) [72].…”