Fast gaze-contingent optimal decompositions for multifocal displays

Abstract

As head-mounted displays (HMDs) commonly present a single, fixed-focus display plane, a conflict can be created between the vergence and accommodation responses of the viewer. Multifocal HMDs have long been investigated as a potential solution in which multiple image planes span the viewer's accommodation range. Such displays require a scene decomposition algorithm to distribute the depiction of objects across image planes, and previous work has shown that simple decompositions can be achieved in real-time. However, recent optimal decompositions further improve image quality, particularly with complex content. Such decompositions are more computationally involved and likely require better alignment of the image planes with the viewer's eyes, which are potential barriers to practical applications. Our goal is to enable interactive optimal decomposition algorithms capable of driving a vergence- and accommodation-tracked multifocal testbed. Ultimately, such a testbed is necessary to establish the requirements for the practical use of multifocal displays, in terms of computational demand and hardware accuracy. To this end, we present an efficient algorithm for optimal decompositions, incorporating insights from vision science. Our method is amenable to GPU implementations and achieves a three-orders-of-magnitude speedup over previous work. We further show that eye tracking can be used for adequate plane alignment with efficient image-based deformations, adjusting for both eye rotation and head movement relative to the display. We also build the first binocular multifocal testbed with integrated eye tracking and accommodation measurement, paving the way to establish practical eye tracking and rendering requirements for this promising class of display. Finally, we report preliminary results from a pilot user study utilizing our testbed, investigating the accommodation response of users to dynamic stimuli presented under optimal decomposition.

Gaze-contingent ocular parallax rendering for virtual reality

“…Eye tracking enables gaze-contingent rendering techniques that adapt effects like magnification, stylization, or geometric level-of-detail to the user's viewpoint [Duchowski et al 2004]. Gaze-contingent rendering is becoming an integral part of modern wearable display systems, enabling techniques such as foveated rendering [Guenter et al 2012;Patney et al 2016] and gaze-contingent varifocal [Dunn et al 2017;Johnson et al 2016;Konrad et al 2015;Liu et al 2008;Padmanaban et al 2017] or multifocal [Akeley et al 2004;Mercier et al 2017;Rolland et al 2000] image displays. Although rendering accommodation-dependent effects, such as chromatic aberrations [Cholewiak et al 2017] and blur at depth edges [Marshall et al 1996;Zannoli et al 2014], have not been directly evaluated with eye-tracked displays, these techniques could be optimized by tracking the user's gaze or accommodation.…”

“…However, none of these works mention or study the effect of ocular parallax. Mercier et al [2017] proposed a multifocal plane display that shifts the decomposed layers according to the tracked pupil position, but because ocular parallax was not explicitly rendered into the displayed images, it was omitted as a visual cue.…”

Towards multifocal displays with dense focal stacks

“…However, their method can only display any given pixel at a single depth. This prohibits the use of rendering techniques [Akeley et al 2004;Mercier et al 2017;Narain et al 2015] that require a pixel to be potentially displayed at many depths with different contents.…”

10‐1: Towards Varifocal Augmented Reality Displays using Deformable Beamsplitter Membranes

“…Early on, Akeley et al [11] demonstrate the benefits of fixed-viewpoint volumetric desktop displays using multiple display planes and generate near-correct focus cues without tracking eye position. Recently such displays were revisited with improved scene decomposition, and gaze-contingent varifocal multi plane capabilities [12], [13]. However, these displays have large power and computational demands with a complex hardware that doesn't lead to a wearable form factor.…”