Towards foveated rendering for gaze-tracked virtual reality

Abstract

Foveated rendering synthesizes images with progressively less detail outside the eye fixation region, potentially unlocking significant speedups for wide field-of-view displays, such as head mounted displays, where target framerate and resolution is increasing faster than the performance of traditional real-time renderers. To study and improve potential gains, we designed a foveated rendering user study to evaluate the perceptual abilities of human peripheral vision when viewing today's displays. We determined that filtering peripheral regions reduces contrast, inducing a sense of tunnel vision. When applying a postprocess contrast enhancement, subjects tolerated up to 2× larger blur radius before detecting differences from a non-foveated ground truth. After verifying these insights on both desktop and head mounted displays augmented with high-speed gaze-tracking, we designed a perceptual target image to strive for when engineering a production foveated renderer. Given our perceptual target, we designed a practical foveated rendering system that reduces number of shades by up to 70% and allows coarsened shading up to 30° closer to the fovea than Guenter et al. [2012] without introducing perceivable aliasing or blur. We filter both pre- and post-shading to address aliasing from undersampling in the periphery, introduce a novel multiresolution- and saccade-aware temporal antialising algorithm, and use contrast enhancement to help recover peripheral details that are resolvable by our eye but degraded by filtering. We validate our system by performing another user study. Frequency analysis shows our system closely matches our perceptual target. Measurements of temporal stability show we obtain quality similar to temporally filtered non-foveated renderings.

Limits of peripheral acuity and implications for VR system design

“…With limited compute power, this results in perceptibly degraded quality. Many techniques have been developed to improve the rendering efficiency in VR, such as multiview stereo rendering, improvements in multisampling, and foveated rendering …”

“…So for a single frame, a “low acuity” and “high acuity” image is generated, possibly with multiple levels in between. These images are then upsampled and composited together into the final frame . Depending on what part of the system the compositing occurs, foveated transmission can lessen bandwidth requirements for internal interfaces between the graphics processing unit and the display panel .…”

“…Many techniques have been developed to improve the rendering efficiency in VR, such as multiview stereo rendering, 5,6 improvements in multisampling, [7][8][9] and foveated rendering. [10][11][12][13][14] Each of these methods reduces the rendering load per frame based on some visual trade-off. Foveated rendering in particular allows computational savings by reducing the total number of pixels needed to be processed.…”

“…These images are then upsampled and composited together into the final frame. 12,13,20 Depending on what part of the system the compositing occurs, foveated transmission can lessen bandwidth requirements for internal interfaces between the graphics processing unit and the display panel. 21 For this approach, image artifacts get introduced at the seams between regions and during upsampling of the lower resolution regions.…”

Z2 traversal order: An interleaving approach for VR stereo rendering on tile-based GPUs

“…By using eye trackers, this technique lowers image quality in the periphery (outside the fovea) to increase rendering performance. The image quality of the peripheral area can be improved by a radially progressive blur, post-process contrast enhancement, and temporal anti-aliasing [19]. NVIDIA multiresolution shading [16] allows multiple scaled viewports in a single pass, and as a result, the edges of the screen distorted by warping and lens distortion can be rendered at reduced resolution without apparent loss of image quality.…”