Gaze in the Dark: Gaze Estimation in a Low-Light Environment with Generative Adversarial Networks

Abstract

In smart interactive environments, such as digital museums or digital exhibition halls, it is important to accurately understand the user’s intent to ensure successful and natural interaction with the exhibition. In the context of predicting user intent, gaze estimation technology has been considered one of the most effective indicators among recently developed interaction techniques (e.g., face orientation estimation, body tracking, and gesture recognition). Previous gaze estimation techniques, however, are known to be effective only in a controlled lab environment under normal lighting conditions. In this study, we propose a novel deep learning-based approach to achieve a successful gaze estimation under various low-light conditions, which is anticipated to be more practical for smart interaction scenarios. The proposed approach utilizes a generative adversarial network (GAN) to enhance users’ eye images captured under low-light conditions, thereby restoring missing information for gaze estimation. Afterward, the GAN-recovered images are fed into the convolutional neural network architecture as input data to estimate the direction of the user gaze. Our experimental results on the modified MPIIGaze dataset demonstrate that the proposed approach achieves an average performance improvement of 4.53%–8.9% under low and dark light conditions, which is a promising step toward further research.

Exploring the Design Space of Optical See-through AR Head-Mounted Displays to Support First Responders in the Field

“…Low visibility environments where FFs work in could be challenging for gaze interactions due to both recognition issues and fixation issues. For recognition issues, gaze estimation algorithms in low-light conditions [47] should be integrated to AR HMDs. For fixation issues, since FFs are concerned about eye wandering in dark environments without targets to focus on, suitable object augmentations should be generated based on recognition on depth or thermal images to provide fixation targets for the FFs for further gaze interaction.…”

Gaze Estimation Approach Using Deep Differential Residual Network

“…In [ 35 ], CNN with long short-term memory (LSTM) network is introduced to be able to capture spatial and temporal features from video frames. In [ 36 ], the generative adversarial network is used to enhance the eye image captured under low and dark light conditions. Despite all the advantages of gaze estimation techniques, there are still some challenges that need to be addressed.…”

Design of low cost-remote eye tracker using Pupil Center-Corneal Reflection technique

“…In addition, the reported accuracy could be calculated from the optimized data of the calibration process and could be the standard deviation rather than the average value 1,3 . Currently, there are two gaze estimation methods, named Feature-based and Appearance-based 4 . Whereas the former uses extracted eye features, the latter treats whole eye images as its inputs.…”

“…Whereas the former uses extracted eye features, the latter treats whole eye images as its inputs. The Feature-based consists of three approaches: 3Dmodel-based, Cross-ratio-based, and Regressionbased 4 . The 3D-model-based models the visual axis and the object, and the gaze point is their intersection.…”

“…The Cross-ratio-based is based on the geometry property under perspective. In the Regression-based, eye features are mapped on gaze points via a regression function 4,5 . In 6 , this function is built based on compensating second-order effects rather than mathematically fitting.…”