Optimization-Based Eye Tracking using Deflectometric Information

Abstract

Eye tracking is an important tool with a wide range of applications in Virtual, Augmented, and Mixed Reality (VR/AR/MR) technologies. State-of-the-art eye tracking methods are either reflection-based and track reflections of sparse point light sources, or image-based and exploit 2D features of the acquired eye image. In this work, we attempt to significantly improve reflection-based methods by utilizing pixel-dense deflectometric surface measurements in combination with optimization-based inverse rendering algorithms. Utilizing the known geometry of our deflectometric setup, we develop a differentiable rendering pipeline based on PyTorch3D that simulates a virtual eye under screen illumination. Eventually, we exploit the imagescreen-correspondence information from the captured measurements to find the eye's rotation, translation, and shape parameters with our renderer via gradient descent. In general, our method does not require a specific pattern and can work with ordinary video frames of the main VR/AR/MR screen itself. We demonstrate real-world experiments with evaluated mean relative gaze errors below 0.45 • at a precision better than 0.11 • . Moreover, we show an improvement of 6X over a representative reflection-based state-of-the-art method in simulation.

Accurate Eye-Tracking from Deflectometric Information using Deep Learning

“…We then evaluate the mean relative error ε0 at each rotation position a w.r.t. the 0° position (see [2] for more details). The evaluation result in Tab.…”

“…A robust, accurate, and fast solution to eye-tracking has widespread applications, spanning from psychology and neuroscience research to Virtual Reality (VR). "Imagebased" eye-tracking methods calculate gaze direction by analyzing features extracted from 2D eye images [1][2][3]. Yet, the limited density of detected eye features constrains the accuracy of gaze direction estimation to a few degrees.…”

“…Yet, the limited density of detected eye features constrains the accuracy of gaze direction estimation to a few degrees. "Reflection-based" methods (e.g., 'glint tracking') exploit cornea reflections from a few point light sources [1][2][3]. These methods typically can reach higher accuracy, but their performance is limited by the limited number of point source reflections (~12 in the current state-of-the-art).…”

“…These methods typically can reach higher accuracy, but their performance is limited by the limited number of point source reflections (~12 in the current state-of-the-art). Recently, our group has introduced a series of approaches that reconstruct the gaze direction from dense deflectometric information [1,2]. By observing the reflection of a full screen (>1 Mio dense pixels), the number of point reflections from the eye surface can be significantly increased (compared to ~12 sparse glints), which translates to an accurate evaluation of the gaze vector.…”

“…The 'Classic Stereo Deflectometry method' [1] delivers the highest accuracy (gaze error <0.25°) but requires two cameras to achieve the best performance. Our 'Optimization-based inverse rendering approach' [2] uses only one camera but exploits prior knowledge about the measured eye to provide a good starting value for the optimization process. In this contribution, we introduce a novel flavor of our deflectometric eye tracking research: We train a Convolutional Neural Network (CNN) model to predict the gaze direction from the captured deflectometry correspondence maps.…”