Gaze Prediction in Dynamic 360° Immersive Videos

Xu, Yanyu; Dong, Yuan; Wu, Junru; Sun, Zhengzhong; Shi, Zhanbiao; Yu, Jingyi; Gao, Shenghua

doi:10.1109/cvpr.2018.00559

Cited by 218 publications

(127 citation statements)

References 24 publications

Supporting

Mentioning

127

Contrasting

Order By: Relevance

“…However, these approaches in [4], [5] use naive models and ignore video content's relation to future movement, thus are less accurate. Other existing works such as in [6]- [10] combine both the video content features and orientation of HMD to predict the future head movement. In [6], the authors use a pre-trained saliency model to predict head movement.…”

Section: A Related Workmentioning

confidence: 99%

“…Although the works in [6]- [10] use both video saliency and history head orientation to predict future head movement, these prior works do not directly treat video frame content in much detail. Since VR videos contain various scenes, and further, each scene has different regions of interest for users.…”

Section: A Related Workmentioning

confidence: 99%

“…Despite tremendous efforts devoted to the prediction of user's viewport in 360°videos [4]- [10], there are no works considering both head movement and content of the video for predicting future viewport of the video. By taking both head movement and video content into account, the VR system can be designed to outperform previous works in predicting the viewport of a 360°video for the user.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Deep Learning for Content-Based Personalized Viewport Prediction of 360-Degree VR Videos

2020

View full text Add to dashboard Cite

In this paper, the problem of head movement prediction for virtual reality videos is studied. In the considered model, a deep learning network is introduced to leverage position data as well as video frame content to predict future head movement. For optimizing data input into this neural network, data sample rate, reduced data, and long-period prediction length are also explored for this model. Simulation results show that the proposed approach yields 16.1% improvement in terms of prediction accuracy compared to a baseline approach that relies only on the position data.

show abstract

Section: A Related Workmentioning

confidence: 99%

Section: A Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Deep Learning for Content-Based Personalized Viewport Prediction of 360-Degree VR Videos

2020

View full text Add to dashboard Cite

show abstract

“…Compared with our setting, their prediction horizon is very short: only 100 ∼ 500 ms. [3] proposed a fixation prediction network that concurrently leverages past FoV locations and video content features to predict the FoV trajectory or tile-based viewing probability maps in the next n frames. In [4], an LSTM is used to encode the history of the FoV scan path and the hidden state features are combined with the visual features to do prediction up to 1 second ahead. A more recent work [5] proposed two deep reinforcement learning models: one offline model is first used to estimate the heatmap of potential FoV for each frame based on the visual features only, an online model is then used to predict the head movement based on the past observed head locations as well as the heatmaps from the offline model.…”

Section: Related Workmentioning

confidence: 99%

“…In related works such as [2] [4], future FoVs are predicted by unrolling a single trained LSTM model. However, such single LSTM model is appropriate only if the input data types and data distribution in the past are the same as that in the future.…”

Section: A Prediction Based On User's Own Pastmentioning

confidence: 99%

Very Long Term Field of View Prediction for 360-Degree Video Streaming

Zhang

Liu

et al. 2019

2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR)

View full text Add to dashboard Cite

360-degree videos have gained increasing popularity in recent years with the developments and advances in Virtual Reality (VR) and Augmented Reality (AR) technologies. In such applications, a user only watches a video scene within a field of view (FoV) centered in a certain direction. Predicting the future FoV in a long time horizon (more than seconds ahead) can help save bandwidth resources in on-demand video streaming while minimizing video freezing in networks with significant bandwidth variations. In this work, we treat the FoV prediction as a sequence learning problem, and propose to predict the target user's future FoV not only based on the user's own past FoV center trajectory but also other users' future FoV locations. We propose multiple prediction models based on two different FoV representations: one using FoV center trajectories and another using equirectangular heatmaps that represent the FoV center distributions. Extensive evaluations with two public datasets demonstrate that the proposed models can significantly outperform benchmark models, and other users' FoVs are very helpful for improving long-term predictions.Index Terms-virtual reality; 360-degree video streaming; time series prediction; field of view;

show abstract

Hybrid‐360: An adaptive bitrate algorithm for tile‐based 360 video streaming

Yang

Jiang

et al. 2021

Trans Emerging Tel Tech

View full text Add to dashboard Cite

In recent years, virtual reality (VR) has become a popular topic. VR videos, as one of the popular applications, can provide the user with an extraordinary and immersed video viewing experience and are gradually entering the public's vision. However, it has faced specific difficulties in the past few years. VR videos need specific collection and displaying equipment. Delivering VR videos needs extra bandwidth and storage space as they contain more information. Traditional adaptive bitrate (ABR) algorithms are mainly designed for 2D videos, and cannot directly apply to VR videos. The delay must be neglectable and there should be fewer stalls. If not, users can suffer from motion sickness. With the development of relevant technologies, those problems can be solved. Using a first‐class VR camera and a head‐mounted display, VR videos can be easily collected and displayed properly. In this work, Hybrid‐360, a novel tile‐based 360 video ABR algorithm, is proposed. Using reinforcement learning, the algorithm can make proper bitrate decisions for every tile in the VR video based on the network and client status. In addition, a system model is proposed to implement the tile‐based 360 ABR algorithm in the real world. Different from traditional 2D videos, VR videos have special characteristics, which brings the need to redesign quality of experience (QoE) model. Additional QoE factors are added into the QoE model, and the weights of every term are adjusted. Using a novel 360 video simulation platform, Hybrid‐360 is then compared with some classical ABR algorithms. It is found to be capable of balancing the video quality as well as video smoothness and fluentness. Also, the algorithm can provide users with a higher QoE.

show abstract

Gaze Prediction in Dynamic 360° Immersive Videos

Cited by 218 publications

References 24 publications

Deep Learning for Content-Based Personalized Viewport Prediction of 360-Degree VR Videos

Deep Learning for Content-Based Personalized Viewport Prediction of 360-Degree VR Videos

Very Long Term Field of View Prediction for 360-Degree Video Streaming

Hybrid‐360: An adaptive bitrate algorithm for tile‐based 360 video streaming

Contact Info

Product

Resources

About