Selective Timewarp Based on Embedded Motion Vectors for Interactive Cloud Virtual Reality

Abstract

Interactive virtual reality (VR) services such as VR gaming require considerable computing power for rendering higher quality VR images at a high frame rate. Thus offloading such VR processing to a cloud or edge computing entity is promising, but is accompanied by long latency leads for a user to see an image of the past viewport, resulting in prohibitive motion sickness. Time warp is a technique for VR that warps a rendered image before scanning it out to the display to correct for the head movement occurring after the rendering and will play an important role in reducing the perceived latency in cloud/edge VR. However, time warp has to be applied to the image areas excluding the head-locked objects such as head-up display, menu bar, and notification that are intended to have a fixed position on a screen; otherwise, they are shown to judder. In this paper, we propose an algorithm that identifies head-locked objects in encoded VR frames at low computational load with no explicit information of the head-locked objects for selective time warp in the cloud/edge VR environments. First, we make a key observation from testbed experiments that motion vectors embedded in an encoded VR video stream are highly correlated with the user's head motion. Based on this finding, the head-locked object detection algorithm is designed to: 1) find a raw shape of each head-locked object from motion vectors embedded in frames and 2) identify the exact pixel-level head-locked region within a limited search area (along the boundary of each raw shape) by monitoring pixel color changes over frames. In order to achieve even lower computational load, the costly part of the algorithm is activated only when a new head-locked object appears. The experimental study demonstrates that the algorithm is able to detect multiple head-locked objects with zero pixel error under limited computational load.INDEX TERMS Virtual reality, cloud VR, timewarp, image reprojection, head-locked object, motion vector.

Motion-Aware Interplay between WiGig and WiFi for Wireless Virtual Reality

“…Recently, notable solutions such as onAirVR [8] have shown the feasibility of wireless VR using WiFi. The key to the success of this solution is the adoption of the Timewarp technique [9,10], where every received image is reprojected before scan-out according to the latest pose of the user; thus, the inconsistency that the user perceives is minimized. However, WiFi greatly requires compressing VR images to fit them into its limited bandwidth, which results in reduced image quality.…”

Velocity- and Error-Aware Switching of Motion Prediction Models for Cloud Virtual Reality

“…The image reprojection technique (also known as time warping) [7]- [9] mitigates VR motion sickness by reducing inconsistencies between the rendered image and the user's viewport at scan-out. This method adjusts rendered frames to account for head pose changes after the scene is rendered [7].…”

“…A sufficiently large mismatch will lead to unpleasant physiological symptoms due to the lag between the vestibular and visual systems of the human body, commonly referred to as motion sickness [6]. The image reprojection technique [7]- [9]-in which a rendered frame is adjusted to account for head pose changes that occur after the scene is renderedhelps to reduce mismatch and alleviate motion sickness. However, as the end-to-end latency increases, large black borders may begin to appear after reprojection [5], [10], negatively impacting the user's quality of experience (QoE) [11].…”

The Impact of Black Edge Artifact on User Experience for the Interactive Cloud VR Services

“…However, Cloud VR games usually get high latency since the delays of coding, decoding, and network transmission are introduced [3]. The latency will cause an inconsistency between a rendered VR image and the user's viewport when the viewport is changed on a headset's display.…”