Time‐Warped Foveated Rendering for Virtual Reality Headsets

Abstract

Rendering in real time for virtual reality headsets with high user immersion is challenging due to strict framerate constraints as well as due to a low tolerance for artefacts. Eye tracking‐based foveated rendering presents an opportunity to strongly increase performance without loss of perceived visual quality. To this end, we propose a novel foveated rendering method for virtual reality headsets with integrated eye tracking hardware. Our method comprises recycling pixels in the periphery by spatio‐temporally reprojecting them from previous frames. Artefacts and disocclusions caused by this reprojection are detected and re‐evaluated according to a confidence value that is determined by a newly introduced formalized perception‐based metric, referred to as confidence function. The foveal region, as well as areas with low confidence values, are redrawn efficiently, as the confidence value allows for the delicate regulation of hierarchical geometry and pixel culling. Hence, the average primitive processing and shading costs are lowered dramatically. Evaluated against regular rendering as well as established foveated rendering methods, our approach shows increased performance in both cases. Furthermore, our method is not restricted to static scenes and provides an acceleration structure for post‐processing passes.

Foveated Rendering: Motivation, Taxonomy, and Research Directions

“…The discern-ability of the fovealperipheral boundary can be reduced by ensuring a smooth transition in the resolution levels from foveal to the peripheral region [Bastani et al 2017]. Franke et al [2021] andMueller et al [2021] use temporal coherence to reduce the number of shading operations. Franke et al [2021] reproject the peripheral region of the previous frame to the current frame, and only the foveal pixels and the less coherent peripheral pixels are rendered.…”

“…Franke et al [2021] andMueller et al [2021] use temporal coherence to reduce the number of shading operations. Franke et al [2021] reproject the peripheral region of the previous frame to the current frame, and only the foveal pixels and the less coherent peripheral pixels are rendered. They propose to use the exact world space pixel positions of the fragments rather than the depth values to avoid reprojection artifacts.…”

Image Features Influence Reaction Time: A Learned Probabilistic Perceptual Model for Saccade Latency

“…Leveraging the perceptual differences between foveal and peripheral vision also improves interactive computer graphics methods, which forms the basis for gaze-contingent foveated rendering and displays. Foveated methods reduce the complexity of graphics or display in the peripheral visual field, which results in an overall improvement of performance or perceived quality [Franke et al 2021;Guenter et al 2012;Kaplanyan et al 2019;Koskela et al 2019Koskela et al , 2016Meng et al 2018;Patney et al 2016;Polychronakis et al 2021;Sun et al 2017Sun et al , 2020Walton et al 2021;Weier et al 2016].…”

Mitigating Cybersickness in Virtual Reality Systems through Foveated Depth-of-Field Blur

“…Some solutions have recently been proposed to overcome these artifacts. Franke et al proposed temporal foveation built into the rasterization pipeline [44]. They introduce a confidence function based on which they decide whether to re-project the pixels from the previous frame or to redraw them.…”