Low-Cost Eye Gaze Prediction System for Interactive Networked Video Streaming

Abstract

Eye gaze is now used as a content adaptation trigger in interactive media applications, such as customized advertisement in video, and bit allocation in streaming video based on region-of-interest (ROI). The reaction time of a gaze-based networked system, however, is lower-bounded by the network round trip time (RTT). Furthermore, only low-sampling-rate gaze data is available when commonly available webcam is employed for gaze tracking. To realize responsive adaptation of media content even under non-negligible RTT and using common low-cost webcams, we propose a Hidden Markov Model (HMM) based gaze-prediction system that utilizes the visual saliency of the content being viewed. Specifically, our HMM has two states corresponding to two of human's intrinsic gaze behavioral movements, and its model parameters are derived offline via analysis of each video's visual saliency maps. Due to the strong prior of likely gaze locations offered by saliency information, accurate runtime gaze prediction is possible even under large RTT and using common webcam.We demonstrate the applicability of our low-cost gaze prediction system by focusing on ROI-based bit allocation for networked video streaming. To reduce transmission rate of a video stream without degrading viewer's perceived visual quality, we allocate more bits to encode the viewer's current spatial ROI, while devoting fewer bits in other spatial regions. The challenge lies in overcoming the delay between the time a viewer's ROI is detected by gaze tracking, to the time the effected video is encoded, delivered and displayed at the viewer's terminal. To this end, we use our proposed low-cost gaze prediction system to predict future eye gaze locations, so that optimized bit allocation can be performed for future frames. Through extensive subjective testing, we show that bit-rate can be reduced by up to 29% without noticeable visual quality degradation when RTT is as high as 200ms.

Improving Driver Gaze Prediction With Reinforced Attention

“…Appearance-based gaze prediction takes images of human face and annotated eye gaze as input to learn a direct image-to-gaze mapping. Feng et al [14] propose a hidden Markov model based gaze prediction system that utilizes the visual saliency of the content being viewed. Davies et al [15] present a multicue gaze prediction framework for open signed video content, and inverstigate which cues are relevant for gaze prediction.…”

Bandwidth-Efficient Live Virtual Reality Streaming Scheme for Reducing View Adaptation Delay

“…Afterwards, allocating different sizes of information (i.e., allocating different encoding rates) have been considered in VR streaming services. This concept was already introduced and widely adopted in medical services and military services with bandwidth limited situation as Region-of-interest (ROI) video coding [16][17][18], which disproportionately allocates larger information rate resources to the ROI as compared to the background to meet the limited bandwidth requirement. The work in [10] suggested smoothing the top and the bottom regions of video before encoding by exploiting characteristic of the equirectangular mapping.…”

On the possibility of predicting gaze aversion to improve video-chat efficiency

“…For telecommunications applications, there have been studies of gaze in multi-party scenarios and the question of who a participant will be looking at. More detailed gaze modeling has also been done, but so far only for non-interactive video [9], [10], [11], [12].…”