EyeNeRF

Abstract

A unique challenge in creating high-quality animatable and relightable 3D avatars of real people is modeling human eyes, particularly in conjunction with the surrounding periocular face region. The challenge of synthesizing eyes is multifold as it requires 1) appropriate representations for the various components of the eye and the periocular region for coherent viewpoint synthesis, capable of representing diffuse, refractive and highly reflective surfaces, 2) disentangling skin and eye appearance from environmental illumination such that it may be rendered under novel lighting conditions, and 3) capturing eyeball motion and the deformation of the surrounding skin to enable re-gazing. These challenges have traditionally necessitated the use of expensive and cumbersome capture setups to obtain high-quality results, and even then, modeling of the full eye region holistically has remained elusive. We present a novel geometry and appearance representation that enables high-fidelity capture and photorealistic animation, view synthesis and relighting of the eye region using only a sparse set of lights and cameras. Our hybrid representation combines an explicit parametric surface model for the eyeball surface with implicit deformable volumetric representations for the periocular region and the interior of the eye. This novel hybrid model has been designed specifically to address the various parts of that exceptionally challenging facial area - the explicit eyeball surface allows modeling refraction and high frequency specular reflection at the cornea, whereas the implicit representation is well suited to model lower frequency skin reflection via spherical harmonics and can represent non-surface structures such as hair (i.e. eyebrows) or highly diffuse volumetric bodies (i.e. sclera), both of which are a challenge for explicit surface models. Tightly integrating the two representations in a joint framework allows controlled photoreal image synthesis and joint optimization of both the geometry parameters of the eyeball and the implicit neural network in continuous 3D space. We show that for high-resolution close-ups of the human eye, our model can synthesize high-fidelity animated gaze from novel views under unseen illumination conditions, allowing to generate visually rich eye imagery.

ShellNeRF: Learning a Controllable High‐resolution Model of the Eye and Periocular Region

“…We complete the full scene model by combining our novel model for the periocular region with an eyeball model from EyeNeRF [LMM*22]. This parametric eyeball model consists of two spheres smoothly blended into each other as defined by the cornea radius, eyeball radius, and the blending angle.…”

“…3D morphable models have been crucial to automated tracking and animation of eyeballs and periocular region [WBM*16], with special emphasis on tracking eyelid contours [WXLY17]. The periocular skin region exhibits complex deformations; data‐driven techniques from multi‐view captures have successfully achieved accurate reconstruction using either mesh [KADM22] or volumetric representations [LMM*22], with limited capacity to animate by interpolating between captured training gazes/expressions. [SWW*20, CSK*22] generate plausible gaze and periocular animations by conditioning their underlying face mesh and implicit volume respectively on the gaze direction for egocentric cameras in VR applications.…”

“…They rely on dense data from thousands of frames from slow‐moving videos with slowly‐changing expressions to successfully constrain this optimization. [LMM*22] combine such a dynamic implicit volume with an explicit eyeball mesh that models ray‐surface interactions such as corneal reflection and refraction to achieve photorealistic view‐synthesis and relighting of the eye region. While such methods work well for performance capture and may interpolate within the training frames, they do not provide any means to control or drive more general expression animations.…”

RelightableHands: Efficient Neural Relighting of Articulated Hand Models

“…While their approach enables efficient relighting for real-time animation using a teacher-student framework, we observe that the bottleneck lighting encoding without visibility information in their student model leads to severe overfitting when applied to hand relighting. EyeNeRF [24] enables the joint learning of geometry and relightable appearance of a moving eyeball model. Compared to eyes, hands exhibit significantly more diverse pose variations, making explicit visibility incorporation essential.…”

EM-Gaze: eye context correlation and metric learning for gaze estimation

“…Lemley et al [ 29 ] further improved the efficiency of gaze estimation with the simplified convolutional neural network on low-quality devices. EyeNeRF provides an efficient method for generating large eye datasets, which may benefit gaze estimation [ 30 ]. Recently, a novel adaptive feature fusion network AFF-Net was proposed [ 4 ].…”