Eye Fixation Assisted Video Saliency Detection via Total Variation-Based Pairwise Interaction

Abstract

As human visual attention is naturally biased towards foreground objects in a scene, it can be used to extract salient objects in video clips. In this work, we proposed a weakly supervised learning based video saliency detection algorithm utilizing eye fixations information from multiple subjects. Our main idea is to extend eye fixations to saliency regions step by step. First, visual seeds are collected using multiple color space geodesic distance based seed region mapping with filtered and extended eye fixations. This operation helps raw fixation points spread to the most likely salient regions, namely, visual seed regions. Second, in order to seize the essential scene structure from video sequences, we introduce the total variance based pairwise interaction model to learn the potential pairwise relationship between foreground and background within a frame or across video frames. In this vein, visual seed regions eventually grow into salient regions. Compared with previous approaches the generated saliency maps has two most outstanding properties: integrity and purity, which are conductive to segment the foreground and significant to the follow-up tasks. Extensive quantitative and qualitative experiments on various video sequences demonstrate that the proposed method outperforms state-of-theart image and video saliency detection algorithms.

Trajectory Saliency Detection Using Consistency-Oriented Latent Codes From a Recurrent Auto-Encoder

“…A second category of video saliency methods aims to predict which areas of the image attract the gaze of human observers. The objective is then to leverage eye-tracking data in order to predict efficiently eye-fixation maps [CBPH16,LZLL14,QGH18].…”

A Study of Eye-Tracking Gaze Point Classification and Application Based on Conditional Random Field

“…The Conditional Random Field (CRF) [15,16] is a discriminative undirected graph model which models the conditional probability of multiple variables after a given observation, in order to solve the problem of sequence labeling. The CRF model often assumes that the observation sequence is x = {x 1 , x 2 , • • • , x n }, and the corresponding class label sequence is y = {y 1 , y 2 , • • • , y n }.…”

“…Coutrot [14] relies on Hidden Markov models (HMM) to classify scan-path fixations and infer an observerrelated characteristic. Qiu [15] proposes Conditional Random Field model (CRF) to classify eye fixation data and Lafferty [16] uses this model CRF to classify sequence data. The same Benfold [17] uses CRF method according head motion, walking direction, and appearance to estimate coarse gaze direction.…”

Trajectory saliency detection using consistency-oriented latent codes from a recurrent auto-encoder

“…A second category of video saliency methods aims to predict which areas of the image attract the gaze of human observers. The objective is then to leverage eyetracking data in order to predict efficiently eye-fixation maps [CBPH16,LZLL14,QGH18].…”