Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

Xu, Mai; Song, Yuhang; Wang, Jianyi; Qiao, Maoying; Huo, Liangyu; Wang, Zulin

doi:10.1109/tpami.2018.2858783

Cited by 170 publications

(97 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In practice, both the HM and EM data are not available for objective VQA on omnidirectional video. Therefore, the predicted HM maps [48] and EM maps [23] are used in our VQA approach. Also, we test the performance of our approach with the ground truth HM and EM data, to show the upper bound performance of our approach.…”

Section: Discussionmentioning

confidence: 99%

Bridge the Gap Between VQA and Human Behavior on Omnidirectional Video

et al. 2018

Proceedings of the 26th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

Omnidirectional video enables spherical stimuli with the 360 × 180 • viewing range. Meanwhile, only the viewport region of omnidirectional video can be seen by the observer through head movement (HM), and an even smaller region within the viewport can be clearly perceived through eye movement (EM). Thus, the subjective quality of omnidirectional video may be correlated with HM and EM of human behavior. To fill in the gap between subjective quality and human behavior, this paper proposes a large-scale visual quality assessment (VQA) dataset of omnidirectional video, called VQA-OV, which collects 60 reference sequences and 540 impaired sequences. Our VQA-OV dataset provides not only the subjective quality scores of sequences but also the HM and EM data of subjects. By mining our dataset, we find that the subjective quality of omnidirectional video is indeed related to HM and EM. Hence, we develop a deep learning model, which embeds HM and EM, for objective VQA on omnidirectional video. Experimental results show that our model significantly improves the state-of-the-art performance of VQA on omnidirectional video. CCS CONCEPTS• Human-centered computing → Virtual reality; KEYWORDSOmnidirectional video, visual quality assessment, human behavior ACM Reference Format:

show abstract

Section: Discussionmentioning

confidence: 99%

Bridge the Gap Between VQA and Human Behavior on Omnidirectional Video

et al. 2018

Proceedings of the 26th ACM International Conference on Multimedia

Self Cite

View full text Add to dashboard Cite

show abstract

“…Furthermore, measuring the distance between states is another important direction to explore, where better representations of states may lead to improvements. Finally, DEHRL may be a promising solution for visual tasks (Xu et al 2018b) with diverse representation and mixed reward functions.…”

Section: Discussionmentioning

confidence: 99%

Diversity-Driven Extensible Hierarchical Reinforcement Learning

Song

Wang

Lukasiewicz

et al. 2019

AAAI

Self Cite

View full text Add to dashboard Cite

Hierarchical reinforcement learning (HRL) has recently shown promising advances on speeding up learning, improving the exploration, and discovering intertask transferable skills. Most recent works focus on HRL with two levels, i.e., a master policy manipulates subpolicies, which in turn manipulate primitive actions. However, HRL with multiple levels is usually needed in many real-world scenarios, whose ultimate goals are highly abstract, while their actions are very primitive. Therefore, in this paper, we propose a diversity-driven extensible HRL (DEHRL), where an extensible and scalable framework is built and learned levelwise to realize HRL with multiple levels. DEHRL follows a popular assumption: diverse subpolicies are useful, i.e., subpolicies are believed to be more useful if they are more diverse. However, existing implementations of this diversity assumption usually have their own drawbacks, which makes them inapplicable to HRL with multiple levels. Consequently, we further propose a novel diversity-driven solution to achieve this assumption in DEHRL. Experimental studies evaluate DEHRL with five baselines from four perspectives in two domains; the results show that DEHRL outperforms the state-of-the-art baselines in all four aspects. arXiv:1811.04324v2 [cs.LG] 13 Nov 2018 Act every 48 steps Level 2 Learned policy at level 2: Fetching ingredients in the order specified by Env Subpolicies at level 0: Actions:Actions the agent can take at each step (moving one leg towards one direction).A list of ingredients to be picked.

show abstract

“…A deep reinforcement learning based viewpoint prediction approach is proposed in [18], in which the reinforcement learning model is established to track the long-term head movement behaviors of humans.…”

Section: Related Workmentioning

confidence: 99%

Probabilistic Tile Visibility-Based Server-Side Rate Adaptation for Adaptive 360-Degree Video Streaming

Liu

Yang

et al. 2020

IEEE J. Sel. Top. Signal Process.

View full text Add to dashboard Cite

In this paper, we study the server-side rate adaptation problem for streaming tile-based adaptive 360-degree videos to multiple users who are competing for transmission resources at the network bottleneck. Specifically, we develop a convolutional neural network (CNN)-based viewpoint prediction model to capture the nonlinear relationship between the future and historical viewpoints. A Laplace distribution model is utilized to characterize the probability distribution of the prediction error. Given the predicted viewpoint, we then map the viewport in the spherical space into its corresponding planar projection in the 2-D plane, and further derive the visibility probability of each tile based on the planar projection and the prediction error probability. According to the visibility probability, tiles are classified as viewport, marginal and invisible tiles. The server-side tile rate allocation problem for multiple users is then formulated as a non-linear discrete optimization problem to minimize the overall received video distortion of all users and the quality difference between the viewport and marginal tiles of each user, subject to the transmission capacity constraints and users' specific viewport requirements.We develop a steepest descent algorithm to solve this non-linear discrete optimization problem, by initializing the feasible starting point in accordance with the optimal solution of its continuous relaxation.Extensive experimental results show that the proposed algorithm can achieve a near-optimal solution, and outperforms the existing rate adaptation schemes for tile-based adaptive 360-video streaming. Index Terms360-degree video, tile-based adaptive streaming, server-side rate adaptation, viewpoint/viewport prediction, tile visibility probability

show abstract

Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

Cited by 170 publications

References 45 publications

Bridge the Gap Between VQA and Human Behavior on Omnidirectional Video

Bridge the Gap Between VQA and Human Behavior on Omnidirectional Video

Diversity-Driven Extensible Hierarchical Reinforcement Learning

Probabilistic Tile Visibility-Based Server-Side Rate Adaptation for Adaptive 360-Degree Video Streaming

Contact Info

Product

Resources

About