A Coarse-to-Fine Adaptive Network for Appearance-Based Gaze Estimation

Abstract

Human gaze is essential for various appealing applications. Aiming at more accurate gaze estimation, a series of recent works propose to utilize face and eye images simultaneously. Nevertheless, face and eye images only serve as independent or parallel feature sources in those works, the intrinsic correlation between their features is overlooked. In this paper we make the following contributions: 1) We propose a coarse-to-fine strategy which estimates a basic gaze direction from face image and refines it with corresponding residual predicted from eye images. 2) Guided by the proposed strategy, we design a framework which introduces a bi-gram model to bridge gaze residual and basic gaze direction, and an attention component to adaptively acquire suitable fine-grained feature. 3) Integrating the above innovations, we construct a coarse-to-fine adaptive network named CA-Net and achieve state-of-the-art performances on MPIIGaze and EyeDiap.

Appearance-based Gaze Estimation With Deep Learning: A Review and Benchmark

“…recent studies propose to use attention mechanism for fusing two eye features. Cheng et al [49] argue that the weights of two eye features are determined by face images due to the specific task in [49], so they assign weights with the guidance of facial features. Bao et al [50] propose a self-attention mechanism to fuse two eye features.…”

“…These two rotations are aggregated into a gaze vector through a gaze transformation layer. Cheng et al [49] propose a coarse-tofine gaze estimation method. They first use a CNN to extract facial features from face images and estimate a basic gaze direction, then they refine the basic gaze direction using eye features.…”

VR content creation and exploration with deep learning: A survey

“…Gaze prediction methods based on user input can be categorized as model-based and appearance-based approaches [142]. Model-based methods fit geometric eye models, detecting eye features using dedicated devices.…”

“…A sub-module of the asymmetric regression network (AR-Net) uses a new asymmetric strategy to estimate both eyes' 3D gaze directions, and a sub-module of the evaluation network (E-Net) evaluates the two eyes' performance to adjust the strategy adaptively during the optimization process. Furthermore, Cheng et al [142] constructed a coarse-to-fine adaptive network named CA-Net. This architecture uses face images to estimate gaze direction, and then predicts corresponding residuals from eye images to refine gaze direction (see Fig.…”

“…gr is used to refine g b to give the output gaze direction g. Reproduced with permission from Ref. [142], c Association for the Advancement of Artificial Intelligence 2020.…”