Very Long Term Field of View Prediction for 360-Degree Video Streaming

Abstract

360-degree videos have gained increasing popularity in recent years with the developments and advances in Virtual Reality (VR) and Augmented Reality (AR) technologies. In such applications, a user only watches a video scene within a field of view (FoV) centered in a certain direction. Predicting the future FoV in a long time horizon (more than seconds ahead) can help save bandwidth resources in on-demand video streaming while minimizing video freezing in networks with significant bandwidth variations. In this work, we treat the FoV prediction as a sequence learning problem, and propose to predict the target user's future FoV not only based on the user's own past FoV center trajectory but also other users' future FoV locations. We propose multiple prediction models based on two different FoV representations: one using FoV center trajectories and another using equirectangular heatmaps that represent the FoV center distributions. Extensive evaluations with two public datasets demonstrate that the proposed models can significantly outperform benchmark models, and other users' FoVs are very helpful for improving long-term predictions.Index Terms-virtual reality; 360-degree video streaming; time series prediction; field of view;

SVP: Sinusoidal Viewport Prediction for 360-Degree Video Streaming

“…Petrangeli et al [11] first identify user clusters with a kind of spectral clustering algorithm, then fit a regression model for each cluster, finally predict with the regression model from the user's corresponding cluster. Li et al [12] utilize LSTM and Attentive Mixture Experts (AME) techniques to train a model to predict based on both the target user's historical fixations and other users' fixations. Nasrabadi et al [13] first cluster users based on their quaternion rotations, and then classify the target user to the corresponding cluster and estimate the future fixation as the cluster center.…”

Low-latency FoV-adaptive Coding and Streaming for Interactive 360° Video Streaming

“…FoV prediction is critical to the performance of FoV-adaptive streaming. In the past, linear regression, weighted linear regression, and truncated linear prediction [5,17,18] as well as neural network based methods [2,6,12] have been proposed. Most of these methods can predict the short-term FoV (within the future 1 second) well with an accuracy of more than 90% [17][12] [6].…”

Human-centric Quality Management of Immersive Multimedia Applications

“…As such,the major part of the available resources can be allocated to this particular part of the video hemisphere, resulting in the perceived quality being optimized. Two major types of viewport prediction can be identified: content-based [22] and content-agnostic [23] pre-…”

“…Content-based prediction aims at characterizing the given content, independent from the user, in terms of saliency maps, motion detection, Regions-Of-Interest (ROIs) etc. Based on this information, density functions can be created that estimate the probability of a generic user looking at a particular part of the hemisphere at a given time instant [22]. Contentagnostic approaches, on the other hand, do not take the content into consideration, but try to predict the user's future fixation point based on this and other user's historical movement.…”